Jul 28 17:21:01.148918 ip-10-0-131-29 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 17:21:01.148928 ip-10-0-131-29 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 17:21:01.148936 ip-10-0-131-29 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 17:21:01.149173 ip-10-0-131-29 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 17:21:10.602820 ip-10-0-131-29 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 17:21:10.602836 ip-10-0-131-29 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ca22f25a73214ad28d0ce9b122d8b0c3 -- Jul 28 17:22:46.718089 ip-10-0-131-29 systemd[1]: Starting Kubernetes Kubelet... Jul 28 17:22:47.168998 ip-10-0-131-29 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:22:47.168998 ip-10-0-131-29 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 17:22:47.168998 ip-10-0-131-29 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:22:47.168998 ip-10-0-131-29 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 17:22:47.168998 ip-10-0-131-29 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:22:47.170101 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.170014 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 17:22:47.175343 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175319 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175347 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175352 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175356 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175363 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175367 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175371 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175375 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175379 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175384 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175388 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175391 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175396 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175400 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:22:47.175398 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175404 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175409 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175414 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175426 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175430 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175435 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175440 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175444 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175449 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175453 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175457 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175462 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175466 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175470 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175477 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175481 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175485 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175487 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175490 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:22:47.175737 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175502 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175505 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175507 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175510 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175512 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175515 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175517 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175520 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175522 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175525 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175529 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175535 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175538 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175541 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175545 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175547 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175550 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175553 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175556 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175559 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:22:47.176211 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175562 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175564 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175567 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175570 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175572 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175575 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175578 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175580 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175583 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175586 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175588 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175593 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175596 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175598 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175601 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175604 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175607 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175609 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175612 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175615 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:22:47.176698 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175619 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175623 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175627 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175630 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175634 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175641 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175645 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175649 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175652 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175655 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175660 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175663 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175667 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175671 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175676 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175678 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175681 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175686 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175689 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:22:47.177271 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175693 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175698 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175701 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175704 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175708 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175711 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.175714 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176565 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176573 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176576 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176579 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176582 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176586 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176588 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176591 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176594 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176597 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176600 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176602 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:22:47.178013 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176605 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176607 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176610 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176613 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176615 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176618 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176622 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176624 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176627 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176629 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176632 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176634 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176637 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176639 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176642 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176644 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176647 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176650 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176653 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176655 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:22:47.178521 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176658 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176660 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176663 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176665 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176668 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176670 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176672 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176675 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176679 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176681 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176683 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176686 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176691 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176695 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176698 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176701 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176704 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176707 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176711 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176714 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:22:47.179039 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176717 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176719 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176722 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176724 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176728 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176730 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176733 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176735 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176738 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176741 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176744 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176747 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176749 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176752 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176771 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176776 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176779 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176781 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176784 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176788 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:22:47.179534 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176790 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176794 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176797 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176802 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176805 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176808 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176811 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176814 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176816 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176819 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176821 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176825 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176828 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176832 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176835 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176838 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176841 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176843 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176846 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:22:47.180056 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176848 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176851 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176853 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176856 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176859 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176861 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176864 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.176866 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180026 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180045 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180050 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180055 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180059 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180065 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180072 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180076 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180082 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180086 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180091 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:22:47.180535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180096 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180104 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180108 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180112 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180117 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180121 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180125 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180129 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180133 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180139 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180146 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180151 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180156 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180166 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180170 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180174 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180178 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180181 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180185 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180191 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:22:47.181060 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180195 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180198 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180204 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180208 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180212 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180220 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180224 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180227 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180231 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180235 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180239 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180243 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180247 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180251 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180254 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180258 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180262 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180267 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180274 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180278 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:22:47.181549 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180282 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180286 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180290 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180294 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180297 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180301 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180305 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180309 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180313 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180318 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180321 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180330 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180335 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180340 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180348 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180352 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180356 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180360 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180365 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:22:47.182089 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180369 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180373 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180377 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180384 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180389 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180393 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180397 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180401 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180405 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180408 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180413 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180417 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180421 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180424 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180428 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180435 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180439 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180444 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180447 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180452 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:22:47.182530 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180457 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180461 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180465 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180469 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180473 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180476 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180480 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180485 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.180493 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180712 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180732 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180740 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180745 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180751 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180768 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180907 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180935 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180939 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180943 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180947 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180951 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180954 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 17:22:47.183028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180958 2564 flags.go:64] FLAG: --cgroup-root="" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180961 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180964 2564 flags.go:64] FLAG: --client-ca-file="" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180967 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180970 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180976 2564 flags.go:64] FLAG: --cluster-domain="" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180979 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180982 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180985 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180988 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180993 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180996 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.180999 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181002 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181005 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181009 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181012 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181017 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181020 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181025 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181028 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181031 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181034 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181037 2564 flags.go:64] FLAG: --enable-server="true" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181040 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 17:22:47.183572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181046 2564 flags.go:64] FLAG: --event-burst="100" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181050 2564 flags.go:64] FLAG: --event-qps="50" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181053 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181056 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181060 2564 flags.go:64] FLAG: --eviction-hard="" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181064 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181067 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181071 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181074 2564 flags.go:64] FLAG: --eviction-soft="" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181077 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181080 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181083 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181087 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181090 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181093 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181096 2564 flags.go:64] FLAG: --feature-gates="" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181100 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181103 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181106 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181109 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181112 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181116 2564 flags.go:64] FLAG: --help="false" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181119 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181122 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 17:22:47.184223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181125 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181128 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181133 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181136 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181139 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181142 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181145 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181148 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181151 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181156 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181159 2564 flags.go:64] FLAG: --kube-reserved="" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181161 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181165 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181168 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181171 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181174 2564 flags.go:64] FLAG: --lock-file="" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181177 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181180 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181183 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181189 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181192 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181195 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181197 2564 flags.go:64] FLAG: --logging-format="text" Jul 28 17:22:47.184909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181200 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181203 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181206 2564 flags.go:64] FLAG: --manifest-url="" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181209 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181215 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181218 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181223 2564 flags.go:64] FLAG: --max-pods="110" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181226 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181229 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181232 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181235 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181238 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181240 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181243 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181251 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181254 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181257 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181260 2564 flags.go:64] FLAG: --pod-cidr="" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181263 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181268 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181271 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181275 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181278 2564 flags.go:64] FLAG: --port="10250" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181281 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 17:22:47.185489 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181284 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e3a6a75aff18676f" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181287 2564 flags.go:64] FLAG: --qos-reserved="" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181290 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181293 2564 flags.go:64] FLAG: --register-node="true" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181296 2564 flags.go:64] FLAG: --register-with-taints="" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181300 2564 flags.go:64] FLAG: --registry-burst="10" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181303 2564 flags.go:64] FLAG: --registry-qps="5" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181307 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181310 2564 flags.go:64] FLAG: --reserved-memory="" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181314 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181317 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181322 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181325 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181328 2564 flags.go:64] FLAG: --runonce="false" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181331 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181334 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181337 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181340 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181343 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181347 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181350 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181353 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181356 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181359 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181362 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181365 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 17:22:47.186135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181369 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181372 2564 flags.go:64] FLAG: --system-cgroups="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181375 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181382 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181384 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181388 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181392 2564 flags.go:64] FLAG: --tls-min-version="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181395 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181398 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181401 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181407 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181410 2564 flags.go:64] FLAG: --v="2" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181414 2564 flags.go:64] FLAG: --version="false" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181419 2564 flags.go:64] FLAG: --vmodule="" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181423 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.181427 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181553 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181558 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181561 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181564 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181567 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181569 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181572 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:22:47.186891 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181575 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181578 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181582 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181586 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181589 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181591 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181594 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181597 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181599 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181602 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181605 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181607 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181610 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181613 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181615 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181619 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181623 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181626 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181629 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:22:47.187456 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181633 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181635 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181638 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181640 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181643 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181652 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181655 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181658 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181661 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181663 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181666 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181668 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181671 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181674 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181676 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181679 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181683 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181685 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181688 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:22:47.187960 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181691 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181693 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181696 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181699 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181701 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181704 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181706 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181709 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181711 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181714 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181717 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181719 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181722 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181726 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181729 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181731 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181734 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181737 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181739 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181750 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:22:47.188427 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181752 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181769 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181773 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181776 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181779 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181783 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181787 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181792 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181796 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181801 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181804 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181807 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181810 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181813 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181816 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181819 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181822 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181825 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181827 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181830 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:22:47.188924 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181833 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181835 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181838 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181841 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181843 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181847 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181850 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181853 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181855 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181858 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181860 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181863 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181874 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.181877 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:22:47.189411 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.182601 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:22:47.190387 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.190361 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 17:22:47.190387 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.190387 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 17:22:47.190453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.190408 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 17:22:47.190453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.190415 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190483 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190489 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190492 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190495 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190498 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190501 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190504 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190507 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:22:47.190507 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190510 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190513 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190516 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190518 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190521 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190524 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190527 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190529 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190532 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190535 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190538 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190540 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190543 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190546 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190548 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190551 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190554 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190558 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190560 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:22:47.190723 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190563 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190566 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190569 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190571 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190574 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190577 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190579 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190582 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190584 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190586 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190589 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190592 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190594 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190597 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190600 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190603 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190606 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190609 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190612 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190614 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:22:47.191225 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190617 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190620 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190623 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190625 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190628 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190630 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190635 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190639 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190642 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190644 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190647 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190651 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190654 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190657 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190659 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190662 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190664 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190667 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190669 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:22:47.191721 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190672 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190676 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190679 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190682 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190685 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190688 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190690 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190694 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190697 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190700 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190702 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190705 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190707 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190710 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190713 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190716 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190718 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190721 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190724 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:22:47.192264 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190727 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190731 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190734 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190737 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190740 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190743 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190747 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190749 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190767 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190772 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190775 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190780 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190793 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190797 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:22:47.192732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.190830 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190967 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190972 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190975 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190978 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190981 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190983 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190986 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190990 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190993 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190995 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.190998 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191001 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191003 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191006 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191009 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191012 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191014 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191017 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191019 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:22:47.193191 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191022 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191024 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191026 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191029 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191031 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191034 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191037 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191040 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191042 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191045 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191048 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191050 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191052 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191055 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191058 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191060 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191063 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191066 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191069 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191072 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:22:47.193681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191075 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191077 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191080 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191083 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191085 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191088 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191090 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191093 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191096 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191098 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191101 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191103 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191106 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191108 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191111 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191113 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191116 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191118 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191122 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191124 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:22:47.194196 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191127 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191129 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191132 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191134 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191137 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191139 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191142 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191145 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191147 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191151 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191155 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191157 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191161 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191164 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191167 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191170 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191172 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191175 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191178 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:22:47.194681 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191181 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191184 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191188 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191191 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191193 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191196 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191198 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191201 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191203 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191206 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191208 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191211 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191222 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191225 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191229 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191232 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191235 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191238 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191241 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:22:47.195192 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191243 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:22:47.195684 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:47.191246 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:22:47.195684 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.191253 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:22:47.195684 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.191393 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 17:22:47.195802 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.195770 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 17:22:47.197539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.197524 2564 server.go:1019] "Starting client certificate rotation" Jul 28 17:22:47.197666 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.197649 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:22:47.197695 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.197683 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:22:47.200596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.200579 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:22:47.202102 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.202082 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:22:47.210437 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.210415 2564 log.go:25] "Validated CRI v1 runtime API" Jul 28 17:22:47.213057 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.213038 2564 log.go:25] "Validated CRI v1 image API" Jul 28 17:22:47.214016 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.214000 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 17:22:47.218287 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.218258 2564 fs.go:135] Filesystem UUIDs: map[4a35bfac-7ad0-49af-b5e6-62b7bad478e8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b3de12f9-c26e-42f9-bd8d-793a3b0baa6b:/dev/nvme0n1p3] Jul 28 17:22:47.218397 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.218286 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 17:22:47.225491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.225351 2564 manager.go:216] Machine: {Timestamp:2026-07-28 17:22:47.223854212 +0000 UTC m=+0.376354397 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116321 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dbf732931946006873a3c3a2012cc SystemUUID:ec2dbf73-2931-9460-0687-3a3c3a2012cc BootID:ca22f25a-7321-4ad2-8d0c-e9b122d8b0c3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:89:9b:76:aa:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:89:9b:76:aa:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:9b:96:f3:4b:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 17:22:47.225491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.225473 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 17:22:47.225735 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.225597 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 17:22:47.229474 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.229440 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 17:22:47.229690 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.229480 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 17:22:47.230433 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.230420 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 17:22:47.230487 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.230440 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 17:22:47.230487 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.230451 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:22:47.230487 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.230473 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:22:47.232170 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.232146 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:22:47.264596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.264561 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 17:22:47.267771 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.267738 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:22:47.267954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.267941 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 17:22:47.268021 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.267987 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 28 17:22:47.268021 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.268010 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 17:22:47.268110 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.268032 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 17:22:47.268110 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.268047 2564 kubelet.go:398] "Adding apiserver pod source" Jul 28 17:22:47.268110 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.268062 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 17:22:47.268486 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.268460 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:22:47.268486 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.268483 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:22:47.271576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.271555 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 17:22:47.274047 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.274030 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 17:22:47.274122 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.274060 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 17:22:47.276116 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276100 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 17:22:47.276116 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276118 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276126 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276135 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276149 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276158 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276167 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276173 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276180 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276186 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276195 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 17:22:47.276230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.276201 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 17:22:47.278076 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.278063 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 17:22:47.278076 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.278075 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 17:22:47.279891 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.279868 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 17:22:47.280537 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.280522 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6gzjw" Jul 28 17:22:47.281300 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.281286 2564 server.go:1268] "Started kubelet" Jul 28 17:22:47.281394 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.281362 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 17:22:47.281615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.281564 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 17:22:47.281735 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.281714 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 17:22:47.282248 ip-10-0-131-29 systemd[1]: Started Kubernetes Kubelet. Jul 28 17:22:47.284066 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.284054 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 28 17:22:47.285172 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.285155 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 17:22:47.290199 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.290176 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 17:22:47.291591 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.291560 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 17:22:47.291856 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.291836 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 17:22:47.292406 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.291229 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-29.ec2.internal.18c683ef92bcd4da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-29.ec2.internal,UID:ip-10-0-131-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-29.ec2.internal,},FirstTimestamp:2026-07-28 17:22:47.281251546 +0000 UTC m=+0.433751714,LastTimestamp:2026-07-28 17:22:47.281251546 +0000 UTC m=+0.433751714,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-29.ec2.internal,}" Jul 28 17:22:47.294300 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.294207 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6gzjw" Jul 28 17:22:47.295391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.295371 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 17:22:47.295484 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.295372 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 17:22:47.295540 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.295484 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.296229 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296212 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 17:22:47.296315 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296234 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 17:22:47.296315 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296264 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 17:22:47.296414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296359 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 28 17:22:47.296414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296368 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 28 17:22:47.296414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296373 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 17:22:47.296414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296385 2564 factory.go:55] Registering systemd factory Jul 28 17:22:47.296414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296393 2564 factory.go:223] Registration of the systemd container factory successfully Jul 28 17:22:47.296672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296656 2564 factory.go:153] Registering CRI-O factory Jul 28 17:22:47.296672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296671 2564 factory.go:223] Registration of the crio container factory successfully Jul 28 17:22:47.296743 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296691 2564 factory.go:103] Registering Raw factory Jul 28 17:22:47.296743 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.296704 2564 manager.go:1196] Started watching for new ooms in manager Jul 28 17:22:47.297021 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.297010 2564 manager.go:349] Starting recovery of all containers Jul 28 17:22:47.301245 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.300170 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 17:22:47.301245 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.300276 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 17:22:47.309079 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.309055 2564 manager.go:354] Recovery completed Jul 28 17:22:47.313531 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.313515 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.315459 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.315437 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.315555 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.315475 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.315555 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.315489 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.316042 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316026 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 17:22:47.316042 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316040 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 17:22:47.316164 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316056 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.316164 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316059 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:22:47.316806 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316751 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.316806 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316790 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.316806 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.316799 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.318879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.318863 2564 policy_none.go:49] "None policy: Start" Jul 28 17:22:47.318930 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.318885 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 17:22:47.318930 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.318900 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 17:22:47.320970 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.320958 2564 policy_none.go:47] "Start" Jul 28 17:22:47.358804 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.358610 2564 manager.go:337] "Starting Device Plugin manager" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.358841 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.358856 2564 server.go:87] "Starting device plugin registration server" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.359103 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.359113 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.359220 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.359308 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.359316 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.359880 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.359923 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.368843 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.370049 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.370096 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.370122 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.370164 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 17:22:47.378107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.372417 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:22:47.459697 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.459624 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.460591 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.460574 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.460638 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.460607 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.460638 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.460617 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.460705 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.460641 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.469918 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.469888 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.470054 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.469923 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-29.ec2.internal\": node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.470814 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.470793 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal"] Jul 28 17:22:47.470874 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.470864 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.472523 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.472501 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.472632 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.472537 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.472632 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.472551 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.474951 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.474929 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.475053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475035 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.475106 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475074 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.475662 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475642 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.475777 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475668 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.475777 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475681 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.475870 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475775 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.475870 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475802 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.475870 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.475817 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.478039 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.478024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.478099 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.478054 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:22:47.478765 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.478743 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:22:47.478825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.478784 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:22:47.478825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.478794 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:22:47.497504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.497469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ba50c98f2bb83f48068145502e392fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal\" (UID: \"3ba50c98f2bb83f48068145502e392fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.497504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.497503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba50c98f2bb83f48068145502e392fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal\" (UID: \"3ba50c98f2bb83f48068145502e392fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.497707 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.497524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/85cc45522d7d542e0d84295a649d3342-config\") pod \"kube-apiserver-proxy-ip-10-0-131-29.ec2.internal\" (UID: \"85cc45522d7d542e0d84295a649d3342\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.503751 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.503721 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-29.ec2.internal\" not found" node="ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.508353 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.508331 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-29.ec2.internal\" not found" node="ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.524844 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.524816 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.598469 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.598434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ba50c98f2bb83f48068145502e392fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal\" (UID: \"3ba50c98f2bb83f48068145502e392fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.598469 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.598465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba50c98f2bb83f48068145502e392fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal\" (UID: \"3ba50c98f2bb83f48068145502e392fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.598656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.598488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/85cc45522d7d542e0d84295a649d3342-config\") pod \"kube-apiserver-proxy-ip-10-0-131-29.ec2.internal\" (UID: \"85cc45522d7d542e0d84295a649d3342\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.598656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.598524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/85cc45522d7d542e0d84295a649d3342-config\") pod \"kube-apiserver-proxy-ip-10-0-131-29.ec2.internal\" (UID: \"85cc45522d7d542e0d84295a649d3342\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.598656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.598546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba50c98f2bb83f48068145502e392fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal\" (UID: \"3ba50c98f2bb83f48068145502e392fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.598656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.598524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ba50c98f2bb83f48068145502e392fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal\" (UID: \"3ba50c98f2bb83f48068145502e392fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.625670 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.625634 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.726463 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.726380 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.807915 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.807886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.813295 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:47.813269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" Jul 28 17:22:47.827217 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.827186 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:47.927580 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:47.927523 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.028012 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.027942 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.128313 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.128274 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.196401 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.196366 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 17:22:48.196832 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.196538 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:22:48.228635 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.228600 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.302333 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.302233 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:17:47 +0000 UTC" deadline="2028-02-26 11:52:49.583346168 +0000 UTC" Jul 28 17:22:48.302333 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.302268 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13866h30m1.281082382s" Jul 28 17:22:48.329314 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.329277 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.429999 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.429960 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.441664 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:48.441623 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cc45522d7d542e0d84295a649d3342.slice/crio-15121802294e37e0b017ed34f3597124fa43b4c770c9c0ef303f52393453fc5f WatchSource:0}: Error finding container 15121802294e37e0b017ed34f3597124fa43b4c770c9c0ef303f52393453fc5f: Status 404 returned error can't find the container with id 15121802294e37e0b017ed34f3597124fa43b4c770c9c0ef303f52393453fc5f Jul 28 17:22:48.446411 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.446393 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:22:48.530357 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.530317 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.622929 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:48.622884 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba50c98f2bb83f48068145502e392fa.slice/crio-9e8c016f2439dbc53667793d8ee08f4e857f7abe7e53882e6e00e33aac9f2dfa WatchSource:0}: Error finding container 9e8c016f2439dbc53667793d8ee08f4e857f7abe7e53882e6e00e33aac9f2dfa: Status 404 returned error can't find the container with id 9e8c016f2439dbc53667793d8ee08f4e857f7abe7e53882e6e00e33aac9f2dfa Jul 28 17:22:48.630450 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:48.630429 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-29.ec2.internal\" not found" Jul 28 17:22:48.657912 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.657879 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:22:48.665848 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.665819 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:22:48.688967 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.688916 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:22:48.696312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.696284 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" Jul 28 17:22:48.710151 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.710126 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:22:48.711395 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.711380 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" Jul 28 17:22:48.722085 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:48.722061 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:22:49.268978 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.268941 2564 apiserver.go:52] "Watching apiserver" Jul 28 17:22:49.277575 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.277534 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 17:22:49.278014 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.277985 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal","openshift-network-operator/iptables-alerter-np54t","openshift-ovn-kubernetes/ovnkube-node-b648k","kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp","openshift-cluster-node-tuning-operator/tuned-zxmqg","openshift-image-registry/node-ca-fl6jf","openshift-multus/multus-72bsm","openshift-multus/multus-additional-cni-plugins-lr2pj","openshift-multus/network-metrics-daemon-n8rn6","openshift-network-diagnostics/network-check-target-gz5d7","kube-system/global-pull-secret-syncer-vv2ch","kube-system/konnectivity-agent-g5cl4","openshift-dns/node-resolver-g9ssz"] Jul 28 17:22:49.280969 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.280944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.281079 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.281030 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:22:49.283394 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.283370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.286244 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.286223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.286476 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.286445 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jtx85\"" Jul 28 17:22:49.286782 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.286639 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 17:22:49.286985 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.286965 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 17:22:49.289331 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.289008 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.289331 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.289138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 17:22:49.289331 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.289182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 17:22:49.289526 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.289389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 17:22:49.289717 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.289695 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 17:22:49.289818 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.289697 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.290083 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.290064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7647\"" Jul 28 17:22:49.290444 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.290277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.293148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.292657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.293148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.292806 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.293322 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.293295 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zs6l6\"" Jul 28 17:22:49.293660 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.293596 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.295033 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.295013 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.295680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.295663 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.295924 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.295910 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.297133 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.296163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 17:22:49.297133 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.296363 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 17:22:49.297133 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.296997 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmflc\"" Jul 28 17:22:49.297447 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.297425 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.299270 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.299094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.300300 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.299651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.300300 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.299707 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z9dgz\"" Jul 28 17:22:49.300957 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.300938 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.301224 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.300942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 17:22:49.302016 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.301996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.302106 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.302037 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 17:22:49.302410 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.302365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 17:22:49.302478 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.302367 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rk7lk\"" Jul 28 17:22:49.302677 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.302658 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.303657 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.303641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.304267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.304249 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.304361 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.304251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 17:22:49.304535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.304518 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.304650 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.304533 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bmc4t\"" Jul 28 17:22:49.306769 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.306736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-slash\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-env-overrides\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-registration-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-host\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-run\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bqc\" (UniqueName: \"kubernetes.io/projected/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-kube-api-access-q9bqc\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9msz\" (UniqueName: \"kubernetes.io/projected/669bb6ec-59b7-47cc-b345-43db24f5c868-kube-api-access-z9msz\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01084945-8e46-41a8-a4eb-75d6ab25b232-hosts-file\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.307982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-var-lib-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.307999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-cnibin\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovnkube-config\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-tmp\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-socket-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysctl-d\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-var-lib-kubelet\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-kubelet\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-run-netns\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysconfig\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-system-cni-dir\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d-konnectivity-ca\") pod \"konnectivity-agent-g5cl4\" (UID: \"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d\") " pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-systemd\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcl4x\" (UniqueName: \"kubernetes.io/projected/2b2becf2-702b-4344-93bb-f18eab272dd5-kube-api-access-jcl4x\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.308535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-sys-fs\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-systemd\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-lib-modules\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmcd\" (UniqueName: \"kubernetes.io/projected/01084945-8e46-41a8-a4eb-75d6ab25b232-kube-api-access-wqmcd\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d-agent-certs\") pod \"konnectivity-agent-g5cl4\" (UID: \"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d\") " pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f424t\"" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-ovn\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-modprobe-d\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-node-log\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-sys\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-run-ovn-kubernetes\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-cni-bin\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlr8\" (UniqueName: \"kubernetes.io/projected/671ae23e-e53e-48fa-a415-41815b5bdfb6-kube-api-access-wqlr8\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.309364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-os-release\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysctl-conf\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01084945-8e46-41a8-a4eb-75d6ab25b232-tmp-dir\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-systemd-units\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-etc-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqxx\" (UniqueName: \"kubernetes.io/projected/3f080b61-36f6-49a2-a7fa-a46634153158-kube-api-access-lxqxx\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.308964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-kubernetes\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-tuned\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-cni-netd\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovn-node-metrics-cert\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovnkube-script-lib\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-log-socket\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309157 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-device-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309372 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309571 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309790 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 17:22:49.310226 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.309887 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gk5j7\"" Jul 28 17:22:49.311120 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.310109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:49.311120 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.310186 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:22:49.311845 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.311788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.311932 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.311868 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:22:49.316298 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.316272 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:22:49.335991 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.335952 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bsc28" Jul 28 17:22:49.344336 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.344313 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bsc28" Jul 28 17:22:49.375359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.375301 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" event={"ID":"3ba50c98f2bb83f48068145502e392fa","Type":"ContainerStarted","Data":"9e8c016f2439dbc53667793d8ee08f4e857f7abe7e53882e6e00e33aac9f2dfa"} Jul 28 17:22:49.376555 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.376526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" event={"ID":"85cc45522d7d542e0d84295a649d3342","Type":"ContainerStarted","Data":"15121802294e37e0b017ed34f3597124fa43b4c770c9c0ef303f52393453fc5f"} Jul 28 17:22:49.398241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.398211 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 17:22:49.409429 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysctl-conf\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01084945-8e46-41a8-a4eb-75d6ab25b232-tmp-dir\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-host\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-systemd-units\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-etc-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqxx\" (UniqueName: \"kubernetes.io/projected/3f080b61-36f6-49a2-a7fa-a46634153158-kube-api-access-lxqxx\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.409617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-kubernetes\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-tuned\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e9fd2df-50cb-436f-9221-d0785c61b25a-iptables-alerter-script\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysctl-conf\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-kubernetes\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01084945-8e46-41a8-a4eb-75d6ab25b232-tmp-dir\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.410053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.409984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-systemd-units\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-cni-netd\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-etc-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410109 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-cni-netd\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovn-node-metrics-cert\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovnkube-script-lib\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-etc-kubernetes\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-log-socket\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410371 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-device-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-cni-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-slash\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-env-overrides\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-registration-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-host\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-netns\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-kubelet\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410608 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-hostroot\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8627n\" (UniqueName: \"kubernetes.io/projected/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-kube-api-access-8627n\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-run\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bqc\" (UniqueName: \"kubernetes.io/projected/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-kube-api-access-q9bqc\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9msz\" (UniqueName: \"kubernetes.io/projected/669bb6ec-59b7-47cc-b345-43db24f5c868-kube-api-access-z9msz\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-host\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01084945-8e46-41a8-a4eb-75d6ab25b232-hosts-file\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.410830 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-device-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-cni-binary-copy\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-slash\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrtr\" (UniqueName: \"kubernetes.io/projected/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-kube-api-access-qrrtr\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-var-lib-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-cnibin\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e9fd2df-50cb-436f-9221-d0785c61b25a-host-slash\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-os-release\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-cni-bin\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovnkube-config\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-tmp\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-conf-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bfc7e4f-dad6-476a-9246-ac413de3d950-dbus\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-socket-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-env-overrides\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.411534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-registration-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovnkube-script-lib\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysctl-d\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411579 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-cnibin\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-var-lib-kubelet\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-var-lib-openvswitch\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-kubelet\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.410518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-log-socket\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-run-netns\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-run-netns\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysconfig\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysctl-d\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-var-lib-kubelet\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411934 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-kubelet\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-system-cni-dir\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.411993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d-konnectivity-ca\") pod \"konnectivity-agent-g5cl4\" (UID: \"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d\") " pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.412359 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-systemd\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcl4x\" (UniqueName: \"kubernetes.io/projected/2b2becf2-702b-4344-93bb-f18eab272dd5-kube-api-access-jcl4x\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-daemon-config\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-sysconfig\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-run\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-k8s-cni-cncf-io\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-cni-multus\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-serviceca\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-sys-fs\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-systemd\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-lib-modules\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmcd\" (UniqueName: \"kubernetes.io/projected/01084945-8e46-41a8-a4eb-75d6ab25b232-kube-api-access-wqmcd\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.412436 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.413104 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.412624 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:49.912592702 +0000 UTC m=+3.065092860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:49.413682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-socket-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.413682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01084945-8e46-41a8-a4eb-75d6ab25b232-hosts-file\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.413682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.413682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4vh\" (UniqueName: \"kubernetes.io/projected/1e9fd2df-50cb-436f-9221-d0785c61b25a-kube-api-access-cm4vh\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.413682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d-agent-certs\") pod \"konnectivity-agent-g5cl4\" (UID: \"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d\") " pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.413682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-ovn\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.414622 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.412998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-modprobe-d\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.414727 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.414665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-system-cni-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.414727 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.414710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-socket-dir-parent\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.414866 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.414745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bfc7e4f-dad6-476a-9246-ac413de3d950-kubelet-config\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.415197 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.415173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovnkube-config\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.415351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.415332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.415413 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.415395 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-system-cni-dir\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.415981 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.415956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-node-log\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.416081 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.415994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d-konnectivity-ca\") pod \"konnectivity-agent-g5cl4\" (UID: \"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d\") " pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.416155 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-systemd\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.416264 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-lib-modules\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.416317 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.416597 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.416666 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-sys\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.416718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:49.416718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-tuned\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.416845 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2becf2-702b-4344-93bb-f18eab272dd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.416968 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.416937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-sys-fs\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.417278 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-systemd\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-tmp\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f080b61-36f6-49a2-a7fa-a46634153158-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-run-ovn\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-etc-modprobe-d\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.417907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-sys\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-run-ovn-kubernetes\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-cni-bin\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlr8\" (UniqueName: \"kubernetes.io/projected/671ae23e-e53e-48fa-a415-41815b5bdfb6-kube-api-access-wqlr8\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-os-release\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-cnibin\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-multus-certs\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.418983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-cni-bin\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.419030 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-host-run-ovn-kubernetes\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.419290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/671ae23e-e53e-48fa-a415-41815b5bdfb6-node-log\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.419326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b2becf2-702b-4344-93bb-f18eab272dd5-os-release\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.421541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.419643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/671ae23e-e53e-48fa-a415-41815b5bdfb6-ovn-node-metrics-cert\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.423934 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.423211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmcd\" (UniqueName: \"kubernetes.io/projected/01084945-8e46-41a8-a4eb-75d6ab25b232-kube-api-access-wqmcd\") pod \"node-resolver-g9ssz\" (UID: \"01084945-8e46-41a8-a4eb-75d6ab25b232\") " pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.423934 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.423414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d-agent-certs\") pod \"konnectivity-agent-g5cl4\" (UID: \"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d\") " pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.425700 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.425632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqxx\" (UniqueName: \"kubernetes.io/projected/3f080b61-36f6-49a2-a7fa-a46634153158-kube-api-access-lxqxx\") pod \"aws-ebs-csi-driver-node-fwzwp\" (UID: \"3f080b61-36f6-49a2-a7fa-a46634153158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.426576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.426272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9msz\" (UniqueName: \"kubernetes.io/projected/669bb6ec-59b7-47cc-b345-43db24f5c868-kube-api-access-z9msz\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.429182 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.429152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlr8\" (UniqueName: \"kubernetes.io/projected/671ae23e-e53e-48fa-a415-41815b5bdfb6-kube-api-access-wqlr8\") pod \"ovnkube-node-b648k\" (UID: \"671ae23e-e53e-48fa-a415-41815b5bdfb6\") " pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.429301 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.429276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bqc\" (UniqueName: \"kubernetes.io/projected/76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf-kube-api-access-q9bqc\") pod \"tuned-zxmqg\" (UID: \"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf\") " pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.429816 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.429793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcl4x\" (UniqueName: \"kubernetes.io/projected/2b2becf2-702b-4344-93bb-f18eab272dd5-kube-api-access-jcl4x\") pod \"multus-additional-cni-plugins-lr2pj\" (UID: \"2b2becf2-702b-4344-93bb-f18eab272dd5\") " pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.520377 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e9fd2df-50cb-436f-9221-d0785c61b25a-iptables-alerter-script\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.520377 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-etc-kubernetes\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-cni-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-netns\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-kubelet\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-hostroot\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8627n\" (UniqueName: \"kubernetes.io/projected/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-kube-api-access-8627n\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-cni-binary-copy\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrtr\" (UniqueName: \"kubernetes.io/projected/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-kube-api-access-qrrtr\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e9fd2df-50cb-436f-9221-d0785c61b25a-host-slash\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-etc-kubernetes\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-os-release\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e9fd2df-50cb-436f-9221-d0785c61b25a-host-slash\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-cni-bin\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.520871 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-conf-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bfc7e4f-dad6-476a-9246-ac413de3d950-dbus\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-cni-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-daemon-config\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-netns\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-k8s-cni-cncf-io\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520972 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-kubelet\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-cni-multus\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-cni-bin\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-serviceca\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-os-release\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4vh\" (UniqueName: \"kubernetes.io/projected/1e9fd2df-50cb-436f-9221-d0785c61b25a-kube-api-access-cm4vh\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-system-cni-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-k8s-cni-cncf-io\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-socket-dir-parent\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bfc7e4f-dad6-476a-9246-ac413de3d950-kubelet-config\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bfc7e4f-dad6-476a-9246-ac413de3d950-dbus\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:49.521249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-cnibin\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-multus-certs\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-host\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-host\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-var-lib-cni-multus\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.520996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e9fd2df-50cb-436f-9221-d0785c61b25a-iptables-alerter-script\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521505 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-cni-binary-copy\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-daemon-config\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-socket-dir-parent\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-serviceca\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bfc7e4f-dad6-476a-9246-ac413de3d950-kubelet-config\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.522033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-system-cni-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.521125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-hostroot\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.522088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-cnibin\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522770 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.522148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-multus-conf-dir\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522770 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.522184 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:49.522770 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.522211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-host-run-multus-certs\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.522770 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.522249 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:50.022233651 +0000 UTC m=+3.174733823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:49.529208 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.529175 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:22:49.529208 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.529206 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:22:49.529208 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.529220 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:49.529481 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.529280 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:50.029260063 +0000 UTC m=+3.181760219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:49.531929 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.531901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4vh\" (UniqueName: \"kubernetes.io/projected/1e9fd2df-50cb-436f-9221-d0785c61b25a-kube-api-access-cm4vh\") pod \"iptables-alerter-np54t\" (UID: \"1e9fd2df-50cb-436f-9221-d0785c61b25a\") " pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.532580 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.532536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrtr\" (UniqueName: \"kubernetes.io/projected/f1206b64-5e1d-4b68-b012-a4963d2d9ccf-kube-api-access-qrrtr\") pod \"node-ca-fl6jf\" (UID: \"f1206b64-5e1d-4b68-b012-a4963d2d9ccf\") " pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.532884 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.532862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8627n\" (UniqueName: \"kubernetes.io/projected/2ef12920-6002-4d48-8959-4dcc9ad1d7d1-kube-api-access-8627n\") pod \"multus-72bsm\" (UID: \"2ef12920-6002-4d48-8959-4dcc9ad1d7d1\") " pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.597857 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.597819 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:22:49.605926 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.605892 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:22:49.616883 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.616846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9ssz" Jul 28 17:22:49.631464 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.631415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" Jul 28 17:22:49.634882 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.634863 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:22:49.637622 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.637594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" Jul 28 17:22:49.646557 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.646525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" Jul 28 17:22:49.654566 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.654537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fl6jf" Jul 28 17:22:49.664369 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.664339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-np54t" Jul 28 17:22:49.672064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.672038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72bsm" Jul 28 17:22:49.924694 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:49.924596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:49.924863 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.924752 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:49.924863 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:49.924829 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:50.924814893 +0000 UTC m=+4.077315052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:50.025063 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.025007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:50.025253 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.025174 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:50.025253 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.025243 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:51.025227571 +0000 UTC m=+4.177727749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:50.125982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.125941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:50.126150 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.126126 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:22:50.126201 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.126155 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:22:50.126201 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.126171 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:50.126273 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.126230 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:51.126215529 +0000 UTC m=+4.278715684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:50.333182 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.333149 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01084945_8e46_41a8_a4eb_75d6ab25b232.slice/crio-58630a9eb73a879c15200795e41d3cfc2041bec9402f332b2118f05eff834fbe WatchSource:0}: Error finding container 58630a9eb73a879c15200795e41d3cfc2041bec9402f332b2118f05eff834fbe: Status 404 returned error can't find the container with id 58630a9eb73a879c15200795e41d3cfc2041bec9402f332b2118f05eff834fbe Jul 28 17:22:50.339004 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.338967 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1206b64_5e1d_4b68_b012_a4963d2d9ccf.slice/crio-2b97d58b69077b1b685f07d34a81e14df85fab556a8708c125fc572a70df4624 WatchSource:0}: Error finding container 2b97d58b69077b1b685f07d34a81e14df85fab556a8708c125fc572a70df4624: Status 404 returned error can't find the container with id 2b97d58b69077b1b685f07d34a81e14df85fab556a8708c125fc572a70df4624 Jul 28 17:22:50.339817 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.339776 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2becf2_702b_4344_93bb_f18eab272dd5.slice/crio-f5ffb3f4a3766588a2ac533fce92b835913311cf7effbe87bb7452ff619ac079 WatchSource:0}: Error finding container f5ffb3f4a3766588a2ac533fce92b835913311cf7effbe87bb7452ff619ac079: Status 404 returned error can't find the container with id f5ffb3f4a3766588a2ac533fce92b835913311cf7effbe87bb7452ff619ac079 Jul 28 17:22:50.341375 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.341301 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef12920_6002_4d48_8959_4dcc9ad1d7d1.slice/crio-8b6b1b4c8fc0ab79fec0842c23902db5ade56ec6d8be9812741071afa5dcf44a WatchSource:0}: Error finding container 8b6b1b4c8fc0ab79fec0842c23902db5ade56ec6d8be9812741071afa5dcf44a: Status 404 returned error can't find the container with id 8b6b1b4c8fc0ab79fec0842c23902db5ade56ec6d8be9812741071afa5dcf44a Jul 28 17:22:50.342338 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.342315 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eb65e8_36c2_48e4_ab3f_5c35a3e3bcbf.slice/crio-e45b7e7900fffa0a6277f68984ae3334c90c7036558f396d46ed91eb37c82bfa WatchSource:0}: Error finding container e45b7e7900fffa0a6277f68984ae3334c90c7036558f396d46ed91eb37c82bfa: Status 404 returned error can't find the container with id e45b7e7900fffa0a6277f68984ae3334c90c7036558f396d46ed91eb37c82bfa Jul 28 17:22:50.344535 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.344511 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f080b61_36f6_49a2_a7fa_a46634153158.slice/crio-8352e4aff8184b5820e1de5ffd1a1c8fae9e46d5a5e677e8d1af5bfce6800dd3 WatchSource:0}: Error finding container 8352e4aff8184b5820e1de5ffd1a1c8fae9e46d5a5e677e8d1af5bfce6800dd3: Status 404 returned error can't find the container with id 8352e4aff8184b5820e1de5ffd1a1c8fae9e46d5a5e677e8d1af5bfce6800dd3 Jul 28 17:22:50.345406 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.345366 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9a5cf5_5a90_44c8_a3de_ac7b4d24b99d.slice/crio-1bf3a06e02333589e6fca67de7e9da19492b53a2b6ab9fab2a1194e8fae49a0b WatchSource:0}: Error finding container 1bf3a06e02333589e6fca67de7e9da19492b53a2b6ab9fab2a1194e8fae49a0b: Status 404 returned error can't find the container with id 1bf3a06e02333589e6fca67de7e9da19492b53a2b6ab9fab2a1194e8fae49a0b Jul 28 17:22:50.346648 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.346506 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9fd2df_50cb_436f_9221_d0785c61b25a.slice/crio-a2407bf9d873c80b645b01c666cf903bcdc2aac9637890d3d38afdf21e50cb2c WatchSource:0}: Error finding container a2407bf9d873c80b645b01c666cf903bcdc2aac9637890d3d38afdf21e50cb2c: Status 404 returned error can't find the container with id a2407bf9d873c80b645b01c666cf903bcdc2aac9637890d3d38afdf21e50cb2c Jul 28 17:22:50.347328 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.347289 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:17:49 +0000 UTC" deadline="2028-02-06 20:18:42.96716397 +0000 UTC" Jul 28 17:22:50.347328 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.347315 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13394h55m52.61985327s" Jul 28 17:22:50.348674 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:22:50.348269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671ae23e_e53e_48fa_a415_41815b5bdfb6.slice/crio-cc9281e745ac82ea765d61ede361e31483cab2bb70c4200b1348afa365f912e1 WatchSource:0}: Error finding container cc9281e745ac82ea765d61ede361e31483cab2bb70c4200b1348afa365f912e1: Status 404 returned error can't find the container with id cc9281e745ac82ea765d61ede361e31483cab2bb70c4200b1348afa365f912e1 Jul 28 17:22:50.384240 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.384202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72bsm" event={"ID":"2ef12920-6002-4d48-8959-4dcc9ad1d7d1","Type":"ContainerStarted","Data":"8b6b1b4c8fc0ab79fec0842c23902db5ade56ec6d8be9812741071afa5dcf44a"} Jul 28 17:22:50.388547 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.388511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9ssz" event={"ID":"01084945-8e46-41a8-a4eb-75d6ab25b232","Type":"ContainerStarted","Data":"58630a9eb73a879c15200795e41d3cfc2041bec9402f332b2118f05eff834fbe"} Jul 28 17:22:50.390694 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.390665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"cc9281e745ac82ea765d61ede361e31483cab2bb70c4200b1348afa365f912e1"} Jul 28 17:22:50.391773 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.391738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-np54t" event={"ID":"1e9fd2df-50cb-436f-9221-d0785c61b25a","Type":"ContainerStarted","Data":"a2407bf9d873c80b645b01c666cf903bcdc2aac9637890d3d38afdf21e50cb2c"} Jul 28 17:22:50.392709 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.392683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5cl4" event={"ID":"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d","Type":"ContainerStarted","Data":"1bf3a06e02333589e6fca67de7e9da19492b53a2b6ab9fab2a1194e8fae49a0b"} Jul 28 17:22:50.393689 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.393658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" event={"ID":"3f080b61-36f6-49a2-a7fa-a46634153158","Type":"ContainerStarted","Data":"8352e4aff8184b5820e1de5ffd1a1c8fae9e46d5a5e677e8d1af5bfce6800dd3"} Jul 28 17:22:50.394573 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.394554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" event={"ID":"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf","Type":"ContainerStarted","Data":"e45b7e7900fffa0a6277f68984ae3334c90c7036558f396d46ed91eb37c82bfa"} Jul 28 17:22:50.399011 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.398983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerStarted","Data":"f5ffb3f4a3766588a2ac533fce92b835913311cf7effbe87bb7452ff619ac079"} Jul 28 17:22:50.402532 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.402505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fl6jf" event={"ID":"f1206b64-5e1d-4b68-b012-a4963d2d9ccf","Type":"ContainerStarted","Data":"2b97d58b69077b1b685f07d34a81e14df85fab556a8708c125fc572a70df4624"} Jul 28 17:22:50.932429 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:50.932118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:50.932585 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.932298 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:50.932585 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:50.932503 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:52.93248146 +0000 UTC m=+6.084981629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:51.034287 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.033709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:51.034287 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.033888 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:51.034287 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.033951 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:53.03393256 +0000 UTC m=+6.186432719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:51.135738 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.135091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:51.135738 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.135280 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:22:51.135738 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.135301 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:22:51.135738 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.135314 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:51.135738 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.135377 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:53.135357769 +0000 UTC m=+6.287857928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:51.348445 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.348328 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:17:49 +0000 UTC" deadline="2028-02-23 09:56:17.774875364 +0000 UTC" Jul 28 17:22:51.348445 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.348392 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13792h33m26.426487474s" Jul 28 17:22:51.373343 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.373309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:51.373515 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.373444 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:22:51.374059 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.373782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:51.374059 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.373900 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:22:51.374059 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.373934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:51.374059 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:51.374015 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:22:51.430938 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.430897 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ba50c98f2bb83f48068145502e392fa" containerID="f30ec9fe47242828945d829e35f99a788264d06058a16bda62651bba7932cd61" exitCode=0 Jul 28 17:22:51.431099 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.431063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" event={"ID":"3ba50c98f2bb83f48068145502e392fa","Type":"ContainerDied","Data":"f30ec9fe47242828945d829e35f99a788264d06058a16bda62651bba7932cd61"} Jul 28 17:22:51.438490 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:51.438460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" event={"ID":"85cc45522d7d542e0d84295a649d3342","Type":"ContainerStarted","Data":"9b00dfafb745fbbdf0688c53b78677911d5304fce837dd683e2f8df9c35b1bc3"} Jul 28 17:22:52.489444 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:52.489044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" event={"ID":"3ba50c98f2bb83f48068145502e392fa","Type":"ContainerStarted","Data":"324537877a50215625f0401f622ca1b9fc12874979d6c0545a4c9302d3d3a30a"} Jul 28 17:22:52.506890 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:52.506825 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal" podStartSLOduration=4.506800445 podStartE2EDuration="4.506800445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:51.474391677 +0000 UTC m=+4.626891858" watchObservedRunningTime="2026-07-28 17:22:52.506800445 +0000 UTC m=+5.659300646" Jul 28 17:22:52.952608 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:52.952518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:52.952799 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:52.952669 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:52.952799 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:52.952735 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:56.952714576 +0000 UTC m=+10.105214755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:53.053515 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:53.053410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:53.053682 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.053626 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:53.053737 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.053689 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:57.053671486 +0000 UTC m=+10.206171644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:53.154851 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:53.154813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:53.155038 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.155021 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:22:53.155096 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.155044 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:22:53.155096 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.155058 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:53.155194 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.155118 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:57.155100033 +0000 UTC m=+10.307600191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:53.371926 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:53.371108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:53.371926 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.371257 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:22:53.371926 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:53.371323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:53.371926 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.371390 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:22:53.371926 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:53.371070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:53.371926 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:53.371827 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:22:55.371248 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:55.371212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:55.371715 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:55.371413 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:22:55.371715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:55.371624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:55.371831 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:55.371736 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:22:55.371831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:55.371211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:55.371914 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:55.371843 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:22:56.991140 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:56.991012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:56.991653 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:56.991192 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:56.991653 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:56.991275 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:04.991254588 +0000 UTC m=+18.143754746 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:22:57.092361 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:57.092320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:57.092600 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.092491 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:57.092600 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.092565 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:05.092545608 +0000 UTC m=+18.245045763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:22:57.193126 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:57.193081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:57.193381 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.193327 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:22:57.193468 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.193390 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:22:57.193468 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.193406 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:57.193567 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.193475 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:05.193454935 +0000 UTC m=+18.345955096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:22:57.371883 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:57.371347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:57.371883 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.371456 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:22:57.374870 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:57.372195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:57.374870 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.372314 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:22:57.374870 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:57.372370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:57.374870 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:57.372448 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:22:59.374786 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:59.374457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:22:59.374786 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:59.374585 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:22:59.375605 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:59.375507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:22:59.375723 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:59.375651 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:22:59.375723 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:22:59.375687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:22:59.375851 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:22:59.375826 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:01.373523 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:01.373491 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:01.374032 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:01.373492 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:01.374032 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:01.373608 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:01.374032 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:01.373492 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:01.374032 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:01.373650 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:01.374032 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:01.373720 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:03.373558 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:03.373503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:03.374049 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:03.373612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:03.374049 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:03.373626 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:03.374049 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:03.373640 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:03.374049 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:03.373704 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:03.374049 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:03.373799 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:05.051863 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:05.051824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:05.052354 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.051975 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:23:05.052354 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.052053 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:21.052030616 +0000 UTC m=+34.204530791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:23:05.152529 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:05.152495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:05.152693 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.152634 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:23:05.152733 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.152694 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:21.152679168 +0000 UTC m=+34.305179327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:23:05.253171 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:05.253135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:05.253348 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.253329 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:23:05.253394 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.253352 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:23:05.253394 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.253362 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:23:05.253491 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.253424 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:21.253406034 +0000 UTC m=+34.405906214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:23:05.371404 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:05.371309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:05.371404 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:05.371349 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:05.371404 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:05.371320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:05.371751 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.371452 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:05.371751 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.371542 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:05.371751 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:05.371653 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:07.371780 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:07.371540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:07.371780 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:07.371654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:07.371780 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:07.371680 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:07.371780 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:07.371656 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:07.372375 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:07.371801 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:07.372375 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:07.371886 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:09.372548 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.371946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:09.372548 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:09.372096 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:09.372548 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.372464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:09.373094 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:09.372566 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:09.373094 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.372599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:09.373094 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:09.372666 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:09.525212 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.524990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fl6jf" event={"ID":"f1206b64-5e1d-4b68-b012-a4963d2d9ccf","Type":"ContainerStarted","Data":"ad4b5ae97f740a75d9f6dbe62a0eba3cc88f40e4ffb4f8f64da9de5d3e4da9ab"} Jul 28 17:23:09.527953 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.527892 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72bsm" event={"ID":"2ef12920-6002-4d48-8959-4dcc9ad1d7d1","Type":"ContainerStarted","Data":"56b711b6ef2973ea68f6781b832f58bf243435df893acf59d811532b4cfbdd08"} Jul 28 17:23:09.531313 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.531171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9ssz" event={"ID":"01084945-8e46-41a8-a4eb-75d6ab25b232","Type":"ContainerStarted","Data":"e58871ac84845a262f545f0ac75a44841bb450c23501b8972fb628c5bbef3ada"} Jul 28 17:23:09.536856 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.536823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"3c4102cb0075d82cb2ba39d179ea3a24f01ed53f305f06207ee402a2a0f2d939"} Jul 28 17:23:09.538542 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.538518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5cl4" event={"ID":"0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d","Type":"ContainerStarted","Data":"d1be5eed9739d4ca6ff44f698491a66f6314360ef76bb5a97410e67a0815c9d5"} Jul 28 17:23:09.540492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.540465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" event={"ID":"3f080b61-36f6-49a2-a7fa-a46634153158","Type":"ContainerStarted","Data":"bb628307698a033a66cf9bf1350c551a853cd82beacbd75356cdf920225a15e6"} Jul 28 17:23:09.542186 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.542114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" event={"ID":"76eb65e8-36c2-48e4-ab3f-5c35a3e3bcbf","Type":"ContainerStarted","Data":"b1f3c72e795f09ceb5958a1378b6fad648409f24e4eaf907e18e1aebad475c24"} Jul 28 17:23:09.545118 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.545068 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-29.ec2.internal" podStartSLOduration=21.545047387 podStartE2EDuration="21.545047387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:52.507436536 +0000 UTC m=+5.659936713" watchObservedRunningTime="2026-07-28 17:23:09.545047387 +0000 UTC m=+22.697547565" Jul 28 17:23:09.546633 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.546602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerStarted","Data":"aa4ca5340a9e9769bc83d161ea67357b73d05bfc976c1b979a6900c524a12f75"} Jul 28 17:23:09.562839 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.562227 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fl6jf" podStartSLOduration=3.596347552 podStartE2EDuration="22.562209644s" totalImagesPullingTime="18.965862092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:09.545396194 +0000 UTC m=+22.697896371" watchObservedRunningTime="2026-07-28 17:23:09.562209644 +0000 UTC m=+22.714709858" Jul 28 17:23:09.588859 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.588814 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g9ssz" podStartSLOduration=3.6191513779999998 podStartE2EDuration="22.588798079s" totalImagesPullingTime="18.969646701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:09.562867537 +0000 UTC m=+22.715367735" watchObservedRunningTime="2026-07-28 17:23:09.588798079 +0000 UTC m=+22.741298256" Jul 28 17:23:09.589364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.589317 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-72bsm" podStartSLOduration=3.582682194 podStartE2EDuration="22.589303848s" totalImagesPullingTime="19.006621654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:09.588699736 +0000 UTC m=+22.741199912" watchObservedRunningTime="2026-07-28 17:23:09.589303848 +0000 UTC m=+22.741804028" Jul 28 17:23:09.610223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.610173 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g5cl4" podStartSLOduration=7.800987334 podStartE2EDuration="22.610156866s" totalImagesPullingTime="14.809169532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:09.609559024 +0000 UTC m=+22.762059227" watchObservedRunningTime="2026-07-28 17:23:09.610156866 +0000 UTC m=+22.762657045" Jul 28 17:23:09.661434 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:09.660875 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zxmqg" podStartSLOduration=3.678149497 podStartE2EDuration="22.660852473s" totalImagesPullingTime="18.982702976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:09.629665821 +0000 UTC m=+22.782166009" watchObservedRunningTime="2026-07-28 17:23:09.660852473 +0000 UTC m=+22.813352651" Jul 28 17:23:10.551370 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.551333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"a0ba487a07a6f3a624d2b0fb2d8ba1672b277d32cdf69e4b491311545c776fba"} Jul 28 17:23:10.552199 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.551436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"ea145272ac759b68b51655925daaa6db334ce80fdac5b880ba49d04f0acb5f67"} Jul 28 17:23:10.552199 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.551480 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"baca4756290c4263df494e65f6482cd99237e302bd0f14ec4c7c9ada2125b895"} Jul 28 17:23:10.552199 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.551513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"fcd8a3dcec29863877ee425aeedf324f0fc6441e5302ad8fb95936b89e8b8bd6"} Jul 28 17:23:10.552199 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.551558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"d4a05db269c1b0c73ae154159ece4459d6eef64f6107ae658903450c6e4d5436"} Jul 28 17:23:10.552749 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.552729 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2becf2-702b-4344-93bb-f18eab272dd5" containerID="aa4ca5340a9e9769bc83d161ea67357b73d05bfc976c1b979a6900c524a12f75" exitCode=0 Jul 28 17:23:10.552859 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.552796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerDied","Data":"aa4ca5340a9e9769bc83d161ea67357b73d05bfc976c1b979a6900c524a12f75"} Jul 28 17:23:10.796836 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:10.796805 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 17:23:11.044098 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.044017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:23:11.044611 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.044581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:23:11.371160 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.370825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:11.371160 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:11.370950 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:11.371160 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.370974 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:11.371160 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:11.371080 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:11.371160 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.371142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:11.371459 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:11.371229 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:11.373460 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.373368 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:23:10.796823778Z","UUID":"8aabd4a0-dabf-493d-b774-5c09ff5b268c","Handler":null,"Name":"","Endpoint":""} Jul 28 17:23:11.375508 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.375484 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 17:23:11.375637 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.375537 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 17:23:11.560402 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.560362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-np54t" event={"ID":"1e9fd2df-50cb-436f-9221-d0785c61b25a","Type":"ContainerStarted","Data":"377b9a85fc40eb454a4e93e3baec0646da263484aee3957e3b3508a18e364d0a"} Jul 28 17:23:11.562908 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.562877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" event={"ID":"3f080b61-36f6-49a2-a7fa-a46634153158","Type":"ContainerStarted","Data":"c4ed8ad145b74395e4ee40507a19991b7d915b34dcdf254156344566d9caf123"} Jul 28 17:23:11.584346 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:11.584285 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-np54t" podStartSLOduration=5.626343319 podStartE2EDuration="24.584265516s" totalImagesPullingTime="18.957922197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:11.58413235 +0000 UTC m=+24.736632528" watchObservedRunningTime="2026-07-28 17:23:11.584265516 +0000 UTC m=+24.736765695" Jul 28 17:23:12.569716 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:12.569625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"3e0ecc250af2a2385122951c10586391c5f46690ede4d32fbc66d8537d1ce046"} Jul 28 17:23:12.571868 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:12.571842 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 17:23:12.572013 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:12.571862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" event={"ID":"3f080b61-36f6-49a2-a7fa-a46634153158","Type":"ContainerStarted","Data":"be64aec1a1bd93c42559c9a4770f2712d396fd09057a37f9932467ec92bccad5"} Jul 28 17:23:12.592949 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:12.592898 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwzwp" podStartSLOduration=4.411051852 podStartE2EDuration="25.59288055s" totalImagesPullingTime="21.181828698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:12.592500107 +0000 UTC m=+25.745000310" watchObservedRunningTime="2026-07-28 17:23:12.59288055 +0000 UTC m=+25.745380727" Jul 28 17:23:13.370698 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:13.370663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:13.370900 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:13.370663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:13.370900 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:13.370815 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:13.370900 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:13.370674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:13.371071 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:13.370892 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:13.371071 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:13.370941 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:15.370909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.370721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:15.371719 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.370729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:15.371719 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:15.371003 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:15.371719 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.370736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:15.371719 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:15.371056 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:15.371719 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:15.371137 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:15.581400 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.581362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" event={"ID":"671ae23e-e53e-48fa-a415-41815b5bdfb6","Type":"ContainerStarted","Data":"b9bf540891f30b0fc05534d48430a299f089ae093f2806b0d452c744cd5c867e"} Jul 28 17:23:15.581722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.581693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:23:15.581855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.581781 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:23:15.583141 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.583116 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2becf2-702b-4344-93bb-f18eab272dd5" containerID="8d3faa05f0fc6057cef3f26b0d1e9e70f65abb4f06f202efebe9f4af469e6816" exitCode=0 Jul 28 17:23:15.583241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.583148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerDied","Data":"8d3faa05f0fc6057cef3f26b0d1e9e70f65abb4f06f202efebe9f4af469e6816"} Jul 28 17:23:15.598549 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.598521 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:23:15.611231 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:15.611180 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" podStartSLOduration=9.562359214 podStartE2EDuration="28.611161715s" totalImagesPullingTime="19.048802501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:15.60947268 +0000 UTC m=+28.761972880" watchObservedRunningTime="2026-07-28 17:23:15.611161715 +0000 UTC m=+28.763661892" Jul 28 17:23:16.289630 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.289296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:23:16.289877 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.289682 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 17:23:16.290016 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.289975 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g5cl4" Jul 28 17:23:16.587898 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.587863 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2becf2-702b-4344-93bb-f18eab272dd5" containerID="c456918a9e1221f919cf580655c5862fa8b77747a80e41e124f9d26929610cea" exitCode=0 Jul 28 17:23:16.588435 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.587949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerDied","Data":"c456918a9e1221f919cf580655c5862fa8b77747a80e41e124f9d26929610cea"} Jul 28 17:23:16.588518 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.588489 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:23:16.603911 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.603885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:23:16.638432 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.638403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vv2ch"] Jul 28 17:23:16.638583 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.638533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:16.638629 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:16.638612 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:16.640581 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.640537 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n8rn6"] Jul 28 17:23:16.640709 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.640653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:16.640802 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:16.640748 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:16.641333 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.641306 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gz5d7"] Jul 28 17:23:16.641422 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:16.641405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:16.641506 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:16.641488 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:18.370590 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:18.370551 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:18.370590 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:18.370589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:18.371153 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:18.370601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:18.371153 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:18.370696 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:18.371153 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:18.370707 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:18.371153 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:18.370794 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:18.595109 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:18.594947 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2becf2-702b-4344-93bb-f18eab272dd5" containerID="dfbaaa7f683fc15761ddc22421fdf299dde539228c7027eb3d20dd6ca0f5333c" exitCode=0 Jul 28 17:23:18.595290 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:18.595008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerDied","Data":"dfbaaa7f683fc15761ddc22421fdf299dde539228c7027eb3d20dd6ca0f5333c"} Jul 28 17:23:20.371055 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:20.370995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:20.371515 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:20.371005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:20.371515 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:20.371173 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8rn6" podUID="669bb6ec-59b7-47cc-b345-43db24f5c868" Jul 28 17:23:20.371515 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:20.371011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:20.371515 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:20.371225 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gz5d7" podUID="b01d1139-a312-4a49-881f-2f6af40c4e55" Jul 28 17:23:20.371515 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:20.371282 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vv2ch" podUID="0bfc7e4f-dad6-476a-9246-ac413de3d950" Jul 28 17:23:21.066029 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:21.065995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:21.066267 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.066158 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:23:21.066267 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.066225 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:53.066209937 +0000 UTC m=+66.218710092 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:23:21.167154 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:21.167117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:21.167352 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.167283 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:23:21.167414 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.167354 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret podName:0bfc7e4f-dad6-476a-9246-ac413de3d950 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:53.167337613 +0000 UTC m=+66.319837771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret") pod "global-pull-secret-syncer-vv2ch" (UID: "0bfc7e4f-dad6-476a-9246-ac413de3d950") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:23:21.267575 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:21.267519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:21.267816 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.267748 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:23:21.267816 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.267789 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:23:21.267816 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.267799 2564 projected.go:196] Error preparing data for projected volume kube-api-access-m88lj for pod openshift-network-diagnostics/network-check-target-gz5d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:23:21.267970 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:21.267853 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj podName:b01d1139-a312-4a49-881f-2f6af40c4e55 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:53.267834974 +0000 UTC m=+66.420335128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88lj" (UniqueName: "kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj") pod "network-check-target-gz5d7" (UID: "b01d1139-a312-4a49-881f-2f6af40c4e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:23:22.152155 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.152084 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-29.ec2.internal" event="NodeReady" Jul 28 17:23:22.152641 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.152242 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 17:23:22.217685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.217652 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-25cpf"] Jul 28 17:23:22.223079 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.223046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.226073 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.226044 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 17:23:22.226627 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.226607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gw5z2\"" Jul 28 17:23:22.229714 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.229686 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 17:23:22.232416 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.232382 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t22pn"] Jul 28 17:23:22.237994 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.237955 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25cpf"] Jul 28 17:23:22.238152 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.238067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.241687 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.241661 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 17:23:22.241940 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.241925 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m7mlr\"" Jul 28 17:23:22.242027 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.241947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 17:23:22.242199 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.242174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 17:23:22.249376 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.249348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t22pn"] Jul 28 17:23:22.371275 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.371237 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:22.371275 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.371275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:22.371506 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.371280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:22.374703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.374669 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:23:22.374855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.374809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:23:22.374855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.374820 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpv4x\"" Jul 28 17:23:22.374998 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.374984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:23:22.375290 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.375173 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rqpml\"" Jul 28 17:23:22.375290 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.375187 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:23:22.377218 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.377129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-config-volume\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.377218 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.377198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.377368 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.377230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0264261c-056b-41f5-a232-c1905715ec4c-cert\") pod \"ingress-canary-t22pn\" (UID: \"0264261c-056b-41f5-a232-c1905715ec4c\") " pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.377368 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.377275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-tmp-dir\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.377368 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.377347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66fc\" (UniqueName: \"kubernetes.io/projected/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-kube-api-access-m66fc\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.377484 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.377421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zg8\" (UniqueName: \"kubernetes.io/projected/0264261c-056b-41f5-a232-c1905715ec4c-kube-api-access-d9zg8\") pod \"ingress-canary-t22pn\" (UID: \"0264261c-056b-41f5-a232-c1905715ec4c\") " pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.478776 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.478674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zg8\" (UniqueName: \"kubernetes.io/projected/0264261c-056b-41f5-a232-c1905715ec4c-kube-api-access-d9zg8\") pod \"ingress-canary-t22pn\" (UID: \"0264261c-056b-41f5-a232-c1905715ec4c\") " pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.478954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.478781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-config-volume\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.478954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.478806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.478954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.478828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0264261c-056b-41f5-a232-c1905715ec4c-cert\") pod \"ingress-canary-t22pn\" (UID: \"0264261c-056b-41f5-a232-c1905715ec4c\") " pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.478954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.478880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-tmp-dir\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.478954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.478912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66fc\" (UniqueName: \"kubernetes.io/projected/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-kube-api-access-m66fc\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.479241 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:22.479217 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:22.479308 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.479292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-tmp-dir\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.479342 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:22.479314 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:23:22.979280331 +0000 UTC m=+36.131780503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:22.479488 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.479466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-config-volume\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.483312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.483289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0264261c-056b-41f5-a232-c1905715ec4c-cert\") pod \"ingress-canary-t22pn\" (UID: \"0264261c-056b-41f5-a232-c1905715ec4c\") " pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.493739 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.493706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66fc\" (UniqueName: \"kubernetes.io/projected/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-kube-api-access-m66fc\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.498195 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.498168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zg8\" (UniqueName: \"kubernetes.io/projected/0264261c-056b-41f5-a232-c1905715ec4c-kube-api-access-d9zg8\") pod \"ingress-canary-t22pn\" (UID: \"0264261c-056b-41f5-a232-c1905715ec4c\") " pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.548985 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.548948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t22pn" Jul 28 17:23:22.752841 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.752673 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t22pn"] Jul 28 17:23:22.755699 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:23:22.755673 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0264261c_056b_41f5_a232_c1905715ec4c.slice/crio-002ec270b8c8e13854d99805f739d83e8dc09c1ffe930389d84109ba7fe32ab4 WatchSource:0}: Error finding container 002ec270b8c8e13854d99805f739d83e8dc09c1ffe930389d84109ba7fe32ab4: Status 404 returned error can't find the container with id 002ec270b8c8e13854d99805f739d83e8dc09c1ffe930389d84109ba7fe32ab4 Jul 28 17:23:22.983750 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:22.983720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:22.983930 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:22.983846 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:22.983930 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:22.983900 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:23:23.98388646 +0000 UTC m=+37.136386615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:23.611678 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:23.611560 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2becf2-702b-4344-93bb-f18eab272dd5" containerID="e58d6fb12c19e9572fdc766e7b2d9512327f49c3525efd7b7e245aa465555dc1" exitCode=0 Jul 28 17:23:23.611678 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:23.611611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerDied","Data":"e58d6fb12c19e9572fdc766e7b2d9512327f49c3525efd7b7e245aa465555dc1"} Jul 28 17:23:23.613019 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:23.612997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t22pn" event={"ID":"0264261c-056b-41f5-a232-c1905715ec4c","Type":"ContainerStarted","Data":"002ec270b8c8e13854d99805f739d83e8dc09c1ffe930389d84109ba7fe32ab4"} Jul 28 17:23:23.990050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:23.989953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:23.990207 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:23.990098 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:23.990207 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:23.990183 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:23:25.990160234 +0000 UTC m=+39.142660409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:24.619351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:24.619312 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2becf2-702b-4344-93bb-f18eab272dd5" containerID="84e9313b04275aaa0193a2a565edaf8f3d5e4bac1161c3d8bf4af6c3ed431f55" exitCode=0 Jul 28 17:23:24.619732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:24.619379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerDied","Data":"84e9313b04275aaa0193a2a565edaf8f3d5e4bac1161c3d8bf4af6c3ed431f55"} Jul 28 17:23:25.623190 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:25.622940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t22pn" event={"ID":"0264261c-056b-41f5-a232-c1905715ec4c","Type":"ContainerStarted","Data":"cc183617326ffd712ffdbc138086a2ab9a796663d93280f4b3539cca3f90d8a7"} Jul 28 17:23:25.626209 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:25.626178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" event={"ID":"2b2becf2-702b-4344-93bb-f18eab272dd5","Type":"ContainerStarted","Data":"4cdd121b43f12571988e0e8a3b6114afe5993287bb0d3e1a35b06dd1474df214"} Jul 28 17:23:25.642346 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:25.642287 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t22pn" podStartSLOduration=1.287474947 podStartE2EDuration="3.642266747s" totalImagesPullingTime="2.3547918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.641922003 +0000 UTC m=+38.794422207" watchObservedRunningTime="2026-07-28 17:23:25.642266747 +0000 UTC m=+38.794766925" Jul 28 17:23:25.665366 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:25.665311 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lr2pj" podStartSLOduration=9.762940547 podStartE2EDuration="38.665293613s" totalImagesPullingTime="28.902353066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.664354694 +0000 UTC m=+38.816854872" watchObservedRunningTime="2026-07-28 17:23:25.665293613 +0000 UTC m=+38.817793799" Jul 28 17:23:26.004508 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:26.004419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:26.004640 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:26.004573 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:26.004684 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:26.004638 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:23:30.004622564 +0000 UTC m=+43.157122718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:30.029922 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:30.029881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:30.030442 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:30.030054 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:30.030442 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:30.030121 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:23:38.03010496 +0000 UTC m=+51.182605118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:38.084290 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:38.084246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:38.084791 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:38.084397 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:38.084791 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:38.084459 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:23:54.084443373 +0000 UTC m=+67.236943528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:48.605960 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:48.605928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b648k" Jul 28 17:23:53.087924 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.087882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:23:53.091047 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.091025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:23:53.098378 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:53.098356 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 17:23:53.098488 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:53.098434 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs podName:669bb6ec-59b7-47cc-b345-43db24f5c868 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:57.098418039 +0000 UTC m=+130.250918194 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs") pod "network-metrics-daemon-n8rn6" (UID: "669bb6ec-59b7-47cc-b345-43db24f5c868") : secret "metrics-daemon-secret" not found Jul 28 17:23:53.188266 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.188221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:53.191573 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.191546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:23:53.203013 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.202980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bfc7e4f-dad6-476a-9246-ac413de3d950-original-pull-secret\") pod \"global-pull-secret-syncer-vv2ch\" (UID: \"0bfc7e4f-dad6-476a-9246-ac413de3d950\") " pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:53.289395 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.289342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vv2ch" Jul 28 17:23:53.289580 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.289469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:53.292997 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.292975 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:23:53.303792 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.303741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:23:53.314440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.314408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88lj\" (UniqueName: \"kubernetes.io/projected/b01d1139-a312-4a49-881f-2f6af40c4e55-kube-api-access-m88lj\") pod \"network-check-target-gz5d7\" (UID: \"b01d1139-a312-4a49-881f-2f6af40c4e55\") " pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:53.419290 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.419260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vv2ch"] Jul 28 17:23:53.422793 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:23:53.422722 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfc7e4f_dad6_476a_9246_ac413de3d950.slice/crio-08a34d9986588ef8100bebc2799c9d9648a219581a200f320431f2014d145255 WatchSource:0}: Error finding container 08a34d9986588ef8100bebc2799c9d9648a219581a200f320431f2014d145255: Status 404 returned error can't find the container with id 08a34d9986588ef8100bebc2799c9d9648a219581a200f320431f2014d145255 Jul 28 17:23:53.598576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.598546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rqpml\"" Jul 28 17:23:53.606285 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.606261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:53.691636 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.691605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vv2ch" event={"ID":"0bfc7e4f-dad6-476a-9246-ac413de3d950","Type":"ContainerStarted","Data":"08a34d9986588ef8100bebc2799c9d9648a219581a200f320431f2014d145255"} Jul 28 17:23:53.733702 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:53.733672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gz5d7"] Jul 28 17:23:53.736847 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:23:53.736816 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01d1139_a312_4a49_881f_2f6af40c4e55.slice/crio-75813abe573e3d95e86d75360d13f82e0e083a12edb417a8f355856b7ccbc2fa WatchSource:0}: Error finding container 75813abe573e3d95e86d75360d13f82e0e083a12edb417a8f355856b7ccbc2fa: Status 404 returned error can't find the container with id 75813abe573e3d95e86d75360d13f82e0e083a12edb417a8f355856b7ccbc2fa Jul 28 17:23:54.094069 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:54.094034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:23:54.094535 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:54.094179 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:23:54.094535 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:54.094242 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls podName:4507d88d-12f1-40c0-8dc1-7d16bb80be4d nodeName:}" failed. No retries permitted until 2026-07-28 17:24:26.094227342 +0000 UTC m=+99.246727499 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls") pod "dns-default-25cpf" (UID: "4507d88d-12f1-40c0-8dc1-7d16bb80be4d") : secret "dns-default-metrics-tls" not found Jul 28 17:23:54.695707 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:54.695668 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gz5d7" event={"ID":"b01d1139-a312-4a49-881f-2f6af40c4e55","Type":"ContainerStarted","Data":"75813abe573e3d95e86d75360d13f82e0e083a12edb417a8f355856b7ccbc2fa"} Jul 28 17:23:57.266101 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.266056 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h"] Jul 28 17:23:57.268943 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.268913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.269453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.269191 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-msvxc"] Jul 28 17:23:57.271783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.271747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.275984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276000 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276074 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276136 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276199 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 17:23:57.276544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276491 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m4psl\"" Jul 28 17:23:57.277034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276744 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 17:23:57.277034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.276935 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wdwn8\"" Jul 28 17:23:57.280935 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.280855 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h"] Jul 28 17:23:57.281061 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.280967 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 17:23:57.284803 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.284704 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-msvxc"] Jul 28 17:23:57.322178 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0bb629-7283-4f68-8e90-c392c834a0e5-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.322178 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txpzt\" (UniqueName: \"kubernetes.io/projected/5b0bb629-7283-4f68-8e90-c392c834a0e5-kube-api-access-txpzt\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.322436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0bb629-7283-4f68-8e90-c392c834a0e5-serving-cert\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.322436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.322436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0bb629-7283-4f68-8e90-c392c834a0e5-service-ca-bundle\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.322436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5b0bb629-7283-4f68-8e90-c392c834a0e5-snapshots\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.322436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/023e50f8-ef13-4635-b097-ef800db19746-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.322686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcbx\" (UniqueName: \"kubernetes.io/projected/023e50f8-ef13-4635-b097-ef800db19746-kube-api-access-6kcbx\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.322686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.322465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b0bb629-7283-4f68-8e90-c392c834a0e5-tmp\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.422912 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.422870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txpzt\" (UniqueName: \"kubernetes.io/projected/5b0bb629-7283-4f68-8e90-c392c834a0e5-kube-api-access-txpzt\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.422937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0bb629-7283-4f68-8e90-c392c834a0e5-serving-cert\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.422969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0bb629-7283-4f68-8e90-c392c834a0e5-service-ca-bundle\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5b0bb629-7283-4f68-8e90-c392c834a0e5-snapshots\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/023e50f8-ef13-4635-b097-ef800db19746-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcbx\" (UniqueName: \"kubernetes.io/projected/023e50f8-ef13-4635-b097-ef800db19746-kube-api-access-6kcbx\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.423105 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:57.423100 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:23:57.423436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b0bb629-7283-4f68-8e90-c392c834a0e5-tmp\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.423436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0bb629-7283-4f68-8e90-c392c834a0e5-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.423436 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:57.423215 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls podName:023e50f8-ef13-4635-b097-ef800db19746 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:57.923197691 +0000 UTC m=+71.075697845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9mw5h" (UID: "023e50f8-ef13-4635-b097-ef800db19746") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:23:57.424386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b0bb629-7283-4f68-8e90-c392c834a0e5-tmp\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.424386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.423893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0bb629-7283-4f68-8e90-c392c834a0e5-service-ca-bundle\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.424386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.424065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5b0bb629-7283-4f68-8e90-c392c834a0e5-snapshots\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.424386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.424338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/023e50f8-ef13-4635-b097-ef800db19746-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.424620 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.424523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0bb629-7283-4f68-8e90-c392c834a0e5-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.425868 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.425845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0bb629-7283-4f68-8e90-c392c834a0e5-serving-cert\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.432405 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.432379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcbx\" (UniqueName: \"kubernetes.io/projected/023e50f8-ef13-4635-b097-ef800db19746-kube-api-access-6kcbx\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.432540 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.432450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txpzt\" (UniqueName: \"kubernetes.io/projected/5b0bb629-7283-4f68-8e90-c392c834a0e5-kube-api-access-txpzt\") pod \"insights-operator-5885fdb759-msvxc\" (UID: \"5b0bb629-7283-4f68-8e90-c392c834a0e5\") " pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.588419 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.588378 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-msvxc" Jul 28 17:23:57.928320 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:57.928225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:57.928487 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:57.928379 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:23:57.928487 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:57.928476 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls podName:023e50f8-ef13-4635-b097-ef800db19746 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:58.928453106 +0000 UTC m=+72.080953279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9mw5h" (UID: "023e50f8-ef13-4635-b097-ef800db19746") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:23:58.240092 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.240061 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-msvxc"] Jul 28 17:23:58.243429 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:23:58.243404 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0bb629_7283_4f68_8e90_c392c834a0e5.slice/crio-39fed34636b928ca5886acce7e05dec3b888908f180699aa1166e7831a47557e WatchSource:0}: Error finding container 39fed34636b928ca5886acce7e05dec3b888908f180699aa1166e7831a47557e: Status 404 returned error can't find the container with id 39fed34636b928ca5886acce7e05dec3b888908f180699aa1166e7831a47557e Jul 28 17:23:58.707847 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.707794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vv2ch" event={"ID":"0bfc7e4f-dad6-476a-9246-ac413de3d950","Type":"ContainerStarted","Data":"2ecc298cfc7a02a161749cfd37ee552d3e24f23337ff0f95f6e8c93a3a64964e"} Jul 28 17:23:58.709088 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.709049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-msvxc" event={"ID":"5b0bb629-7283-4f68-8e90-c392c834a0e5","Type":"ContainerStarted","Data":"39fed34636b928ca5886acce7e05dec3b888908f180699aa1166e7831a47557e"} Jul 28 17:23:58.710543 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.710519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gz5d7" event={"ID":"b01d1139-a312-4a49-881f-2f6af40c4e55","Type":"ContainerStarted","Data":"4af75570322cd20ee73f9faa49fefb76dd0b20804ae4f05f06741542120f1a60"} Jul 28 17:23:58.710693 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.710673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:23:58.728952 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.728899 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vv2ch" podStartSLOduration=67.030588819 podStartE2EDuration="1m11.728876817s" totalImagesPullingTime="4.698287998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:58.728543797 +0000 UTC m=+71.881043974" watchObservedRunningTime="2026-07-28 17:23:58.728876817 +0000 UTC m=+71.881376988" Jul 28 17:23:58.748801 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.748732 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gz5d7" podStartSLOduration=67.36635342 podStartE2EDuration="1m11.748709849s" totalImagesPullingTime="4.382356429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:58.747546307 +0000 UTC m=+71.900046484" watchObservedRunningTime="2026-07-28 17:23:58.748709849 +0000 UTC m=+71.901210025" Jul 28 17:23:58.937024 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:23:58.936992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:23:58.937207 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:58.937162 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:23:58.937255 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:23:58.937242 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls podName:023e50f8-ef13-4635-b097-ef800db19746 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:00.937221369 +0000 UTC m=+74.089721526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9mw5h" (UID: "023e50f8-ef13-4635-b097-ef800db19746") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:00.407114 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.407080 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-shhb8"] Jul 28 17:24:00.410144 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.410127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.415814 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.415788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 17:24:00.415973 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.415788 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 17:24:00.415973 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.415830 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7f8h\"" Jul 28 17:24:00.427235 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.427207 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-shhb8"] Jul 28 17:24:00.446623 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.446594 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.446793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.446690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c428648b-bbbc-434c-9d53-2629cdbd4935-data-volume\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.446793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.446714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c428648b-bbbc-434c-9d53-2629cdbd4935-crio-socket\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.446793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.446729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.446793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.446772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9ts\" (UniqueName: \"kubernetes.io/projected/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-api-access-wj9ts\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547486 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547653 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c428648b-bbbc-434c-9d53-2629cdbd4935-data-volume\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547653 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:00.547602 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:24:00.547717 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c428648b-bbbc-434c-9d53-2629cdbd4935-crio-socket\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547717 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:00.547662 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:01.047647035 +0000 UTC m=+74.200147189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : secret "insights-runtime-extractor-tls" not found Jul 28 17:24:00.547717 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c428648b-bbbc-434c-9d53-2629cdbd4935-crio-socket\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547717 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547717 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9ts\" (UniqueName: \"kubernetes.io/projected/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-api-access-wj9ts\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.547937 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:00.547792 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:24:00.547937 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:00.547857 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:01.047837623 +0000 UTC m=+74.200337797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : configmap "kube-rbac-proxy" not found Jul 28 17:24:00.547999 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.547950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c428648b-bbbc-434c-9d53-2629cdbd4935-data-volume\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.557217 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.557188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9ts\" (UniqueName: \"kubernetes.io/projected/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-api-access-wj9ts\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:00.717747 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.717656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-msvxc" event={"ID":"5b0bb629-7283-4f68-8e90-c392c834a0e5","Type":"ContainerStarted","Data":"2da91ef2cd372e7918f1bd02d53a0d6b1b6e0f0d3447396094bb0fee9e77b25d"} Jul 28 17:24:00.740960 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.740904 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-msvxc" podStartSLOduration=2.278692503 podStartE2EDuration="3.740888028s" totalImagesPullingTime="1.462195525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:00.74010294 +0000 UTC m=+73.892603116" watchObservedRunningTime="2026-07-28 17:24:00.740888028 +0000 UTC m=+73.893388203" Jul 28 17:24:00.954304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:00.951623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:24:00.954304 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:00.951815 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:00.954304 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:00.951886 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls podName:023e50f8-ef13-4635-b097-ef800db19746 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.951866997 +0000 UTC m=+78.104367169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9mw5h" (UID: "023e50f8-ef13-4635-b097-ef800db19746") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:01.052998 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:01.052887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:01.052998 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:01.053001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:01.053225 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:01.053028 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:24:01.053225 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:01.053090 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:24:01.053225 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:01.053094 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.05307702 +0000 UTC m=+75.205577174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : secret "insights-runtime-extractor-tls" not found Jul 28 17:24:01.053225 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:01.053150 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.053134748 +0000 UTC m=+75.205634906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : configmap "kube-rbac-proxy" not found Jul 28 17:24:02.061420 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:02.061375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:02.061815 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:02.061459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:02.061815 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:02.061529 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:24:02.061815 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:02.061589 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.061575645 +0000 UTC m=+77.214075802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : secret "insights-runtime-extractor-tls" not found Jul 28 17:24:02.061815 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:02.061529 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:24:02.061815 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:02.061658 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.061647319 +0000 UTC m=+77.214147494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : configmap "kube-rbac-proxy" not found Jul 28 17:24:02.179250 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:02.179220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9ssz_01084945-8e46-41a8-a4eb-75d6ab25b232/dns-node-resolver/0.log" Jul 28 17:24:03.378737 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:03.378707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fl6jf_f1206b64-5e1d-4b68-b012-a4963d2d9ccf/node-ca/0.log" Jul 28 17:24:04.077815 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:04.077738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:04.077977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:04.077834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:04.077977 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:04.077878 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:24:04.077977 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:04.077944 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:24:04.077977 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:04.077959 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.077943646 +0000 UTC m=+81.230443803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : configmap "kube-rbac-proxy" not found Jul 28 17:24:04.078118 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:04.077982 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.077971228 +0000 UTC m=+81.230471382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : secret "insights-runtime-extractor-tls" not found Jul 28 17:24:04.180057 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:04.180030 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t22pn_0264261c-056b-41f5-a232-c1905715ec4c/serve-healthcheck-canary/0.log" Jul 28 17:24:04.984184 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:04.984143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:24:04.984577 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:04.984301 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:04.984577 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:04.984370 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls podName:023e50f8-ef13-4635-b097-ef800db19746 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:12.984353028 +0000 UTC m=+86.136853182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9mw5h" (UID: "023e50f8-ef13-4635-b097-ef800db19746") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:06.756364 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.756334 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj"] Jul 28 17:24:06.759598 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.759574 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" Jul 28 17:24:06.760316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.760290 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6"] Jul 28 17:24:06.762693 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.762671 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 17:24:06.763023 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.763008 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.763836 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.763813 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2w5rh\"" Jul 28 17:24:06.763925 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.763837 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:24:06.766442 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.766421 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8"] Jul 28 17:24:06.768260 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.768238 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 17:24:06.768344 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.768272 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 17:24:06.768512 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.768497 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:24:06.769128 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.768967 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 17:24:06.769128 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.768947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9d52f\"" Jul 28 17:24:06.773875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.773615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:06.774427 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.774405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj"] Jul 28 17:24:06.776717 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.776687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6"] Jul 28 17:24:06.777000 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.776979 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8k9ss\"" Jul 28 17:24:06.777298 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.777278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:24:06.777482 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.777328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 17:24:06.777548 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.777371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 17:24:06.786555 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.786530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8"] Jul 28 17:24:06.798490 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.798454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqns9\" (UniqueName: \"kubernetes.io/projected/be13f61c-c224-4f9e-b233-f8b9b2d16968-kube-api-access-mqns9\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.798721 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.798699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hl8\" (UniqueName: \"kubernetes.io/projected/b57768de-3577-447d-9cf1-85dc72dc600a-kube-api-access-v7hl8\") pod \"volume-data-source-validator-549c6d54cf-klwbj\" (UID: \"b57768de-3577-447d-9cf1-85dc72dc600a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" Jul 28 17:24:06.798969 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.798777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be13f61c-c224-4f9e-b233-f8b9b2d16968-config\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.798969 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.798824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be13f61c-c224-4f9e-b233-f8b9b2d16968-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.862467 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.862430 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rrhzz"] Jul 28 17:24:06.865574 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.865549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:06.869146 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.869114 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 17:24:06.869974 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.869949 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 17:24:06.869974 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.869961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hv7x2\"" Jul 28 17:24:06.870287 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.869949 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:24:06.870287 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.869978 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 17:24:06.875438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.875414 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 17:24:06.877391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.877372 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rrhzz"] Jul 28 17:24:06.899319 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be13f61c-c224-4f9e-b233-f8b9b2d16968-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.899319 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-trusted-ca\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:06.899600 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:06.899600 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hl8\" (UniqueName: \"kubernetes.io/projected/b57768de-3577-447d-9cf1-85dc72dc600a-kube-api-access-v7hl8\") pod \"volume-data-source-validator-549c6d54cf-klwbj\" (UID: \"b57768de-3577-447d-9cf1-85dc72dc600a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" Jul 28 17:24:06.899600 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ps8\" (UniqueName: \"kubernetes.io/projected/c2e2153a-d371-423d-90f4-2b4aabcb720f-kube-api-access-q6ps8\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:06.899600 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be13f61c-c224-4f9e-b233-f8b9b2d16968-config\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.899600 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-config\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:06.899842 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfj2w\" (UniqueName: \"kubernetes.io/projected/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-kube-api-access-wfj2w\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:06.899842 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqns9\" (UniqueName: \"kubernetes.io/projected/be13f61c-c224-4f9e-b233-f8b9b2d16968-kube-api-access-mqns9\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.899842 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-serving-cert\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:06.900016 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.899998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be13f61c-c224-4f9e-b233-f8b9b2d16968-config\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.901614 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.901583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be13f61c-c224-4f9e-b233-f8b9b2d16968-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.908680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.908646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqns9\" (UniqueName: \"kubernetes.io/projected/be13f61c-c224-4f9e-b233-f8b9b2d16968-kube-api-access-mqns9\") pod \"service-ca-operator-5d6c75f84f-fggn6\" (UID: \"be13f61c-c224-4f9e-b233-f8b9b2d16968\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:06.909186 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:06.909168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hl8\" (UniqueName: \"kubernetes.io/projected/b57768de-3577-447d-9cf1-85dc72dc600a-kube-api-access-v7hl8\") pod \"volume-data-source-validator-549c6d54cf-klwbj\" (UID: \"b57768de-3577-447d-9cf1-85dc72dc600a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" Jul 28 17:24:07.000462 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.000408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-config\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.000669 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.000490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj2w\" (UniqueName: \"kubernetes.io/projected/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-kube-api-access-wfj2w\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.000669 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.000523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-serving-cert\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.000669 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.000573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-trusted-ca\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.000669 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.000621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:07.000669 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.000652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ps8\" (UniqueName: \"kubernetes.io/projected/c2e2153a-d371-423d-90f4-2b4aabcb720f-kube-api-access-q6ps8\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:07.000948 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:07.000748 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 17:24:07.000948 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:07.000870 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls podName:c2e2153a-d371-423d-90f4-2b4aabcb720f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:07.500844558 +0000 UTC m=+80.653344727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls") pod "cluster-samples-operator-6748489656-p62z8" (UID: "c2e2153a-d371-423d-90f4-2b4aabcb720f") : secret "samples-operator-tls" not found Jul 28 17:24:07.001217 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.001186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-config\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.001866 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.001848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-trusted-ca\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.003008 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.002988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-serving-cert\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.010808 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.010709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ps8\" (UniqueName: \"kubernetes.io/projected/c2e2153a-d371-423d-90f4-2b4aabcb720f-kube-api-access-q6ps8\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:07.010808 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.010713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj2w\" (UniqueName: \"kubernetes.io/projected/e25edec6-93b9-4ef7-95a4-31fb4ffcbf22-kube-api-access-wfj2w\") pod \"console-operator-584dcb6b54-rrhzz\" (UID: \"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22\") " pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.071403 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.071350 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" Jul 28 17:24:07.079785 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.079737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" Jul 28 17:24:07.174123 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.174029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:07.208233 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.208155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj"] Jul 28 17:24:07.211163 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:07.211127 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57768de_3577_447d_9cf1_85dc72dc600a.slice/crio-905bb4ea38fac46f8278a7ae6b74a51bb0d5f425d9709c6c9db3c7dd6c19b639 WatchSource:0}: Error finding container 905bb4ea38fac46f8278a7ae6b74a51bb0d5f425d9709c6c9db3c7dd6c19b639: Status 404 returned error can't find the container with id 905bb4ea38fac46f8278a7ae6b74a51bb0d5f425d9709c6c9db3c7dd6c19b639 Jul 28 17:24:07.226575 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.226543 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6"] Jul 28 17:24:07.229720 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:07.229685 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe13f61c_c224_4f9e_b233_f8b9b2d16968.slice/crio-6e2fc256bc92c368c071509aa1b669ba15dc314d3cdda20fec6295aa4d8e406a WatchSource:0}: Error finding container 6e2fc256bc92c368c071509aa1b669ba15dc314d3cdda20fec6295aa4d8e406a: Status 404 returned error can't find the container with id 6e2fc256bc92c368c071509aa1b669ba15dc314d3cdda20fec6295aa4d8e406a Jul 28 17:24:07.320319 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.320288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rrhzz"] Jul 28 17:24:07.325499 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:07.325469 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25edec6_93b9_4ef7_95a4_31fb4ffcbf22.slice/crio-ce6c5d2daffb72d03bf6f8702dcecbabf7cf24014b8c0cb7711461d9179d4f24 WatchSource:0}: Error finding container ce6c5d2daffb72d03bf6f8702dcecbabf7cf24014b8c0cb7711461d9179d4f24: Status 404 returned error can't find the container with id ce6c5d2daffb72d03bf6f8702dcecbabf7cf24014b8c0cb7711461d9179d4f24 Jul 28 17:24:07.504732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.504696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:07.504943 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:07.504871 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 17:24:07.505013 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:07.504955 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls podName:c2e2153a-d371-423d-90f4-2b4aabcb720f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.504934745 +0000 UTC m=+81.657434916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls") pod "cluster-samples-operator-6748489656-p62z8" (UID: "c2e2153a-d371-423d-90f4-2b4aabcb720f") : secret "samples-operator-tls" not found Jul 28 17:24:07.735404 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.735366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" event={"ID":"be13f61c-c224-4f9e-b233-f8b9b2d16968","Type":"ContainerStarted","Data":"6e2fc256bc92c368c071509aa1b669ba15dc314d3cdda20fec6295aa4d8e406a"} Jul 28 17:24:07.736729 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.736686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" event={"ID":"b57768de-3577-447d-9cf1-85dc72dc600a","Type":"ContainerStarted","Data":"905bb4ea38fac46f8278a7ae6b74a51bb0d5f425d9709c6c9db3c7dd6c19b639"} Jul 28 17:24:07.737847 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:07.737819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" event={"ID":"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22","Type":"ContainerStarted","Data":"ce6c5d2daffb72d03bf6f8702dcecbabf7cf24014b8c0cb7711461d9179d4f24"} Jul 28 17:24:08.110779 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:08.110689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:08.111244 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:08.110788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:08.111244 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:08.110789 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:24:08.111244 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:08.110869 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:24:08.111244 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:08.110958 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:16.110935864 +0000 UTC m=+89.263436018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : configmap "kube-rbac-proxy" not found Jul 28 17:24:08.111244 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:08.111021 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:16.111001616 +0000 UTC m=+89.263501774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : secret "insights-runtime-extractor-tls" not found Jul 28 17:24:08.515260 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:08.515227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:08.515425 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:08.515384 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 17:24:08.515481 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:08.515432 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls podName:c2e2153a-d371-423d-90f4-2b4aabcb720f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:10.515417441 +0000 UTC m=+83.667917595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls") pod "cluster-samples-operator-6748489656-p62z8" (UID: "c2e2153a-d371-423d-90f4-2b4aabcb720f") : secret "samples-operator-tls" not found Jul 28 17:24:08.742078 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:08.741982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" event={"ID":"b57768de-3577-447d-9cf1-85dc72dc600a","Type":"ContainerStarted","Data":"b678836e72cb6a2aac6bdcf8960216ba9e6c0014e735e8aeec9733f218e4355a"} Jul 28 17:24:08.760588 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:08.760084 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-klwbj" podStartSLOduration=1.56615297 podStartE2EDuration="2.760062524s" totalImagesPullingTime="1.193909554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:08.759532762 +0000 UTC m=+81.912032939" watchObservedRunningTime="2026-07-28 17:24:08.760062524 +0000 UTC m=+81.912562701" Jul 28 17:24:10.531089 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.531046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:10.531510 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:10.531237 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 17:24:10.531510 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:10.531318 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls podName:c2e2153a-d371-423d-90f4-2b4aabcb720f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:14.531297495 +0000 UTC m=+87.683797653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls") pod "cluster-samples-operator-6748489656-p62z8" (UID: "c2e2153a-d371-423d-90f4-2b4aabcb720f") : secret "samples-operator-tls" not found Jul 28 17:24:10.749220 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.749191 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/0.log" Jul 28 17:24:10.749401 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.749233 2564 generic.go:356] "Generic (PLEG): container finished" podID="e25edec6-93b9-4ef7-95a4-31fb4ffcbf22" containerID="cf87a7a1707818ed3d3e1bd8d8731e52b7fa200161dd32ad819f7a70edc723b9" exitCode=255 Jul 28 17:24:10.749401 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.749270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" event={"ID":"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22","Type":"ContainerDied","Data":"cf87a7a1707818ed3d3e1bd8d8731e52b7fa200161dd32ad819f7a70edc723b9"} Jul 28 17:24:10.749567 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.749539 2564 scope.go:117] "RemoveContainer" containerID="cf87a7a1707818ed3d3e1bd8d8731e52b7fa200161dd32ad819f7a70edc723b9" Jul 28 17:24:10.750853 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.750819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" event={"ID":"be13f61c-c224-4f9e-b233-f8b9b2d16968","Type":"ContainerStarted","Data":"1941e94c097ea0ca3b1deb50df7ed552a0501b25a9fc8733cf60ce73cae7b329"} Jul 28 17:24:10.788672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:10.788624 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" podStartSLOduration=2.2172346960000002 podStartE2EDuration="4.78860788s" totalImagesPullingTime="2.571373184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:10.78828941 +0000 UTC m=+83.940789584" watchObservedRunningTime="2026-07-28 17:24:10.78860788 +0000 UTC m=+83.941108056" Jul 28 17:24:11.755781 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:11.755730 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:24:11.756190 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:11.756173 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/0.log" Jul 28 17:24:11.756255 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:11.756219 2564 generic.go:356] "Generic (PLEG): container finished" podID="e25edec6-93b9-4ef7-95a4-31fb4ffcbf22" containerID="c7b39ff8f6811c8776a7b2448eb6cda6451be983c229d798d3e8af22262c27c1" exitCode=255 Jul 28 17:24:11.756307 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:11.756252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" event={"ID":"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22","Type":"ContainerDied","Data":"c7b39ff8f6811c8776a7b2448eb6cda6451be983c229d798d3e8af22262c27c1"} Jul 28 17:24:11.756361 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:11.756313 2564 scope.go:117] "RemoveContainer" containerID="cf87a7a1707818ed3d3e1bd8d8731e52b7fa200161dd32ad819f7a70edc723b9" Jul 28 17:24:11.756562 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:11.756545 2564 scope.go:117] "RemoveContainer" containerID="c7b39ff8f6811c8776a7b2448eb6cda6451be983c229d798d3e8af22262c27c1" Jul 28 17:24:11.756745 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:11.756725 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-rrhzz_openshift-console-operator(e25edec6-93b9-4ef7-95a4-31fb4ffcbf22)\"" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" podUID="e25edec6-93b9-4ef7-95a4-31fb4ffcbf22" Jul 28 17:24:12.764271 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:12.764244 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:24:12.764620 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:12.764598 2564 scope.go:117] "RemoveContainer" containerID="c7b39ff8f6811c8776a7b2448eb6cda6451be983c229d798d3e8af22262c27c1" Jul 28 17:24:12.764796 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:12.764752 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-rrhzz_openshift-console-operator(e25edec6-93b9-4ef7-95a4-31fb4ffcbf22)\"" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" podUID="e25edec6-93b9-4ef7-95a4-31fb4ffcbf22" Jul 28 17:24:13.055712 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:13.055626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:24:13.055878 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:13.055810 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:13.055916 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:13.055875 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls podName:023e50f8-ef13-4635-b097-ef800db19746 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:29.055860375 +0000 UTC m=+102.208360529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9mw5h" (UID: "023e50f8-ef13-4635-b097-ef800db19746") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:24:14.567710 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:14.567664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:14.568132 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:14.567815 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 17:24:14.568132 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:14.567884 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls podName:c2e2153a-d371-423d-90f4-2b4aabcb720f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:22.567869332 +0000 UTC m=+95.720369485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls") pod "cluster-samples-operator-6748489656-p62z8" (UID: "c2e2153a-d371-423d-90f4-2b4aabcb720f") : secret "samples-operator-tls" not found Jul 28 17:24:16.179743 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:16.179686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:16.180169 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:16.179841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:16.180169 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:16.179897 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:24:16.180169 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:16.179978 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls podName:c428648b-bbbc-434c-9d53-2629cdbd4935 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:32.179957208 +0000 UTC m=+105.332457373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shhb8" (UID: "c428648b-bbbc-434c-9d53-2629cdbd4935") : secret "insights-runtime-extractor-tls" not found Jul 28 17:24:16.180443 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:16.180425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c428648b-bbbc-434c-9d53-2629cdbd4935-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:17.175417 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:17.175364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:17.175606 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:17.175468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:17.175846 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:17.175834 2564 scope.go:117] "RemoveContainer" containerID="c7b39ff8f6811c8776a7b2448eb6cda6451be983c229d798d3e8af22262c27c1" Jul 28 17:24:17.176000 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:17.175981 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-rrhzz_openshift-console-operator(e25edec6-93b9-4ef7-95a4-31fb4ffcbf22)\"" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" podUID="e25edec6-93b9-4ef7-95a4-31fb4ffcbf22" Jul 28 17:24:22.632782 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:22.632721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:22.635029 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:22.635004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2e2153a-d371-423d-90f4-2b4aabcb720f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p62z8\" (UID: \"c2e2153a-d371-423d-90f4-2b4aabcb720f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:22.686562 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:22.686524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" Jul 28 17:24:22.813479 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:22.813444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8"] Jul 28 17:24:23.799358 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:23.799322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" event={"ID":"c2e2153a-d371-423d-90f4-2b4aabcb720f","Type":"ContainerStarted","Data":"b3dfaf6fcca50f0a7796fe693e95f1e8ef4aa2c0344357a3a9df1dc03bd49728"} Jul 28 17:24:24.803465 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:24.803435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" event={"ID":"c2e2153a-d371-423d-90f4-2b4aabcb720f","Type":"ContainerStarted","Data":"c61b54ee2221b2c78d65646bca1d93ac916208b9e9bec6d87038ba71e06232a7"} Jul 28 17:24:25.808091 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:25.808055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" event={"ID":"c2e2153a-d371-423d-90f4-2b4aabcb720f","Type":"ContainerStarted","Data":"f8e5537a7c8b76453332635c517e7b1d4959c625948e4f964fcaab4cfd422d75"} Jul 28 17:24:25.830141 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:25.830088 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p62z8" podStartSLOduration=18.036084342 podStartE2EDuration="19.830071468s" totalImagesPullingTime="1.793987126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:25.828890932 +0000 UTC m=+98.981391127" watchObservedRunningTime="2026-07-28 17:24:25.830071468 +0000 UTC m=+98.982571644" Jul 28 17:24:26.159350 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:26.159309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:24:26.161585 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:26.161565 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4507d88d-12f1-40c0-8dc1-7d16bb80be4d-metrics-tls\") pod \"dns-default-25cpf\" (UID: \"4507d88d-12f1-40c0-8dc1-7d16bb80be4d\") " pod="openshift-dns/dns-default-25cpf" Jul 28 17:24:26.438437 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:26.438350 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gw5z2\"" Jul 28 17:24:26.446048 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:26.446017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25cpf" Jul 28 17:24:26.571788 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:26.571738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25cpf"] Jul 28 17:24:26.574742 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:26.574698 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4507d88d_12f1_40c0_8dc1_7d16bb80be4d.slice/crio-75026a05a4108b7b48871669b5e85f06cbf98ee4abbf6ae593987bfb03cea9e5 WatchSource:0}: Error finding container 75026a05a4108b7b48871669b5e85f06cbf98ee4abbf6ae593987bfb03cea9e5: Status 404 returned error can't find the container with id 75026a05a4108b7b48871669b5e85f06cbf98ee4abbf6ae593987bfb03cea9e5 Jul 28 17:24:26.812613 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:26.812523 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25cpf" event={"ID":"4507d88d-12f1-40c0-8dc1-7d16bb80be4d","Type":"ContainerStarted","Data":"75026a05a4108b7b48871669b5e85f06cbf98ee4abbf6ae593987bfb03cea9e5"} Jul 28 17:24:28.370649 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:28.370615 2564 scope.go:117] "RemoveContainer" containerID="c7b39ff8f6811c8776a7b2448eb6cda6451be983c229d798d3e8af22262c27c1" Jul 28 17:24:28.822976 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:28.822893 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:24:28.822976 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:28.822962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" event={"ID":"e25edec6-93b9-4ef7-95a4-31fb4ffcbf22","Type":"ContainerStarted","Data":"28481cd90de688e1723442cb4b9fba075c50e287716987e510292e06f681a91c"} Jul 28 17:24:28.823381 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:28.823306 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:28.843411 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:28.843348 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" podStartSLOduration=20.364968826 podStartE2EDuration="22.84333103s" totalImagesPullingTime="2.478362204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:28.842501809 +0000 UTC m=+101.995002027" watchObservedRunningTime="2026-07-28 17:24:28.84333103 +0000 UTC m=+101.995831207" Jul 28 17:24:29.084210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.084162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:24:29.086518 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.086488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/023e50f8-ef13-4635-b097-ef800db19746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9mw5h\" (UID: \"023e50f8-ef13-4635-b097-ef800db19746\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:24:29.234414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.234378 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-rrhzz" Jul 28 17:24:29.380962 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.380926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" Jul 28 17:24:29.565937 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.565904 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h"] Jul 28 17:24:29.568582 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:29.568554 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023e50f8_ef13_4635_b097_ef800db19746.slice/crio-d9093949dd037acee6df393878c4a747bf2a0e09d28c0c4f78cc5e7642e6f053 WatchSource:0}: Error finding container d9093949dd037acee6df393878c4a747bf2a0e09d28c0c4f78cc5e7642e6f053: Status 404 returned error can't find the container with id d9093949dd037acee6df393878c4a747bf2a0e09d28c0c4f78cc5e7642e6f053 Jul 28 17:24:29.714694 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.714619 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gz5d7" Jul 28 17:24:29.828574 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.828534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25cpf" event={"ID":"4507d88d-12f1-40c0-8dc1-7d16bb80be4d","Type":"ContainerStarted","Data":"dda69922cdb2fde1ab73ce06f47680346cf3d0b179eac3f6c6a7166f48bbe150"} Jul 28 17:24:29.828732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.828600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25cpf" event={"ID":"4507d88d-12f1-40c0-8dc1-7d16bb80be4d","Type":"ContainerStarted","Data":"61441f18f7a318b50e89e744310d8279543dbc439cf1df2570ff640ba08c444f"} Jul 28 17:24:29.828732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.828628 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-25cpf" Jul 28 17:24:29.829705 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.829681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" event={"ID":"023e50f8-ef13-4635-b097-ef800db19746","Type":"ContainerStarted","Data":"d9093949dd037acee6df393878c4a747bf2a0e09d28c0c4f78cc5e7642e6f053"} Jul 28 17:24:29.847972 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:29.847915 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-25cpf" podStartSLOduration=65.134975233 podStartE2EDuration="1m7.847897334s" totalImagesPullingTime="2.712922101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:29.847871017 +0000 UTC m=+103.000371213" watchObservedRunningTime="2026-07-28 17:24:29.847897334 +0000 UTC m=+103.000397506" Jul 28 17:24:31.837861 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:31.837816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" event={"ID":"023e50f8-ef13-4635-b097-ef800db19746","Type":"ContainerStarted","Data":"db62554b71e5d4e17a9057f45c3c0f2d1213c315c24dea5a6a4d9012163d7f9f"} Jul 28 17:24:31.855793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:31.855692 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9mw5h" podStartSLOduration=32.800280194 podStartE2EDuration="34.855675353s" totalImagesPullingTime="2.055395159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:31.854873272 +0000 UTC m=+105.007373447" watchObservedRunningTime="2026-07-28 17:24:31.855675353 +0000 UTC m=+105.008175507" Jul 28 17:24:32.212584 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:32.212544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:32.214867 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:32.214843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c428648b-bbbc-434c-9d53-2629cdbd4935-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shhb8\" (UID: \"c428648b-bbbc-434c-9d53-2629cdbd4935\") " pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:32.219849 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:32.219823 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-shhb8" Jul 28 17:24:32.365654 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:32.365622 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-shhb8"] Jul 28 17:24:32.369200 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:32.369173 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc428648b_bbbc_434c_9d53_2629cdbd4935.slice/crio-e4c848c67756df51ae06d70bbe17a4df1a5f2d4ff615568d119d8a8a7194788a WatchSource:0}: Error finding container e4c848c67756df51ae06d70bbe17a4df1a5f2d4ff615568d119d8a8a7194788a: Status 404 returned error can't find the container with id e4c848c67756df51ae06d70bbe17a4df1a5f2d4ff615568d119d8a8a7194788a Jul 28 17:24:32.843206 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:32.843166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shhb8" event={"ID":"c428648b-bbbc-434c-9d53-2629cdbd4935","Type":"ContainerStarted","Data":"e4c848c67756df51ae06d70bbe17a4df1a5f2d4ff615568d119d8a8a7194788a"} Jul 28 17:24:35.983254 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:35.983219 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5"] Jul 28 17:24:36.008112 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.008080 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5"] Jul 28 17:24:36.008305 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.008210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.011239 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.011204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zc8cp\"" Jul 28 17:24:36.012615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.012305 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 28 17:24:36.014576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.014494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 28 17:24:36.087654 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.087619 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm"] Jul 28 17:24:36.094549 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.094516 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cc4fb576f-7rttz"] Jul 28 17:24:36.094712 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.094662 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" Jul 28 17:24:36.098117 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.098087 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nwkfx\"" Jul 28 17:24:36.099230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.099202 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-m2kzp"] Jul 28 17:24:36.099356 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.099324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.102795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.102269 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 17:24:36.102795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.102643 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d8jtm\"" Jul 28 17:24:36.102795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.102791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 17:24:36.103031 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.102929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 17:24:36.104972 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.104949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:36.107661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.107636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 17:24:36.108454 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.108433 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 17:24:36.109301 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.109264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm"] Jul 28 17:24:36.110885 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.110854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 17:24:36.111155 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.111114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hzbcf\"" Jul 28 17:24:36.115420 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.115394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cc4fb576f-7rttz"] Jul 28 17:24:36.142474 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.142437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f135a084-91da-49cb-8a7e-60b9601ab0b4-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-zb6h5\" (UID: \"f135a084-91da-49cb-8a7e-60b9601ab0b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.142672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.142556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f135a084-91da-49cb-8a7e-60b9601ab0b4-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-zb6h5\" (UID: \"f135a084-91da-49cb-8a7e-60b9601ab0b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.143702 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.143674 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-m2kzp"] Jul 28 17:24:36.203160 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.203118 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cc4fb576f-7rttz"] Jul 28 17:24:36.203436 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:36.203408 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-gdxk4 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" podUID="c6fe8ebe-f6bd-4e00-a8bd-cce316794565" Jul 28 17:24:36.243132 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-trusted-ca\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243132 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-tls\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243362 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f135a084-91da-49cb-8a7e-60b9601ab0b4-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-zb6h5\" (UID: \"f135a084-91da-49cb-8a7e-60b9601ab0b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.243362 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f135a084-91da-49cb-8a7e-60b9601ab0b4-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-zb6h5\" (UID: \"f135a084-91da-49cb-8a7e-60b9601ab0b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.243362 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxj6\" (UniqueName: \"kubernetes.io/projected/ae12c5de-a7b3-406a-b792-5cf0d2950ebb-kube-api-access-nqxj6\") pod \"network-check-source-7d88cb55d8-qs6hm\" (UID: \"ae12c5de-a7b3-406a-b792-5cf0d2950ebb\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" Jul 28 17:24:36.243362 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-certificates\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243362 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-installation-pull-secrets\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243601 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-image-registry-private-configuration\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243601 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxk4\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-kube-api-access-gdxk4\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243601 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkj6\" (UniqueName: \"kubernetes.io/projected/4783cdd8-1ae0-4520-88b2-c5fc3faf4159-kube-api-access-gzkj6\") pod \"downloads-7c97dc4555-m2kzp\" (UID: \"4783cdd8-1ae0-4520-88b2-c5fc3faf4159\") " pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:36.243601 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-ca-trust-extracted\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243601 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-bound-sa-token\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.243911 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.243887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f135a084-91da-49cb-8a7e-60b9601ab0b4-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-zb6h5\" (UID: \"f135a084-91da-49cb-8a7e-60b9601ab0b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.246102 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.246079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f135a084-91da-49cb-8a7e-60b9601ab0b4-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-zb6h5\" (UID: \"f135a084-91da-49cb-8a7e-60b9601ab0b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.324316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.324280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" Jul 28 17:24:36.344504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.344470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-trusted-ca\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.344504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.344511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-tls\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxj6\" (UniqueName: \"kubernetes.io/projected/ae12c5de-a7b3-406a-b792-5cf0d2950ebb-kube-api-access-nqxj6\") pod \"network-check-source-7d88cb55d8-qs6hm\" (UID: \"ae12c5de-a7b3-406a-b792-5cf0d2950ebb\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-certificates\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-installation-pull-secrets\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-image-registry-private-configuration\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxk4\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-kube-api-access-gdxk4\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkj6\" (UniqueName: \"kubernetes.io/projected/4783cdd8-1ae0-4520-88b2-c5fc3faf4159-kube-api-access-gzkj6\") pod \"downloads-7c97dc4555-m2kzp\" (UID: \"4783cdd8-1ae0-4520-88b2-c5fc3faf4159\") " pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-ca-trust-extracted\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-bound-sa-token\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.345585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-trusted-ca\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.346254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-certificates\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.346991 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.346399 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-ca-trust-extracted\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.347473 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.347441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-tls\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.348465 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.348439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-installation-pull-secrets\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.348705 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.348668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-image-registry-private-configuration\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.358383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.358350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxj6\" (UniqueName: \"kubernetes.io/projected/ae12c5de-a7b3-406a-b792-5cf0d2950ebb-kube-api-access-nqxj6\") pod \"network-check-source-7d88cb55d8-qs6hm\" (UID: \"ae12c5de-a7b3-406a-b792-5cf0d2950ebb\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" Jul 28 17:24:36.359501 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.359471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkj6\" (UniqueName: \"kubernetes.io/projected/4783cdd8-1ae0-4520-88b2-c5fc3faf4159-kube-api-access-gzkj6\") pod \"downloads-7c97dc4555-m2kzp\" (UID: \"4783cdd8-1ae0-4520-88b2-c5fc3faf4159\") " pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:36.359617 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.359474 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxk4\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-kube-api-access-gdxk4\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.359728 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.359705 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-bound-sa-token\") pod \"image-registry-7cc4fb576f-7rttz\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.407881 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.407851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" Jul 28 17:24:36.425612 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.425581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:36.856660 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.856630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:36.868148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:36.868120 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:37.052731 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.052692 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-trusted-ca\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.052747 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-image-registry-private-configuration\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.052804 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-certificates\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.052983 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-bound-sa-token\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053029 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-tls\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053063 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-installation-pull-secrets\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053091 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdxk4\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-kube-api-access-gdxk4\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053134 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-ca-trust-extracted\") pod \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\" (UID: \"c6fe8ebe-f6bd-4e00-a8bd-cce316794565\") " Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:37.053210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053192 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:37.053638 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053392 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-certificates\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.053638 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053409 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-trusted-ca\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.053638 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.053543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:24:37.055687 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.055627 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:24:37.055835 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.055772 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:24:37.055835 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.055808 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:24:37.055944 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.055859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-kube-api-access-gdxk4" (OuterVolumeSpecName: "kube-api-access-gdxk4") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "kube-api-access-gdxk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:24:37.055944 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.055905 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c6fe8ebe-f6bd-4e00-a8bd-cce316794565" (UID: "c6fe8ebe-f6bd-4e00-a8bd-cce316794565"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:24:37.154553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.154459 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-ca-trust-extracted\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.154553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.154494 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-image-registry-private-configuration\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.154553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.154507 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-bound-sa-token\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.154553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.154519 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-registry-tls\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.154553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.154532 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-installation-pull-secrets\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.154553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.154544 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxk4\" (UniqueName: \"kubernetes.io/projected/c6fe8ebe-f6bd-4e00-a8bd-cce316794565-kube-api-access-gdxk4\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:24:37.439539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.439383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm"] Jul 28 17:24:37.453313 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.453180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-m2kzp"] Jul 28 17:24:37.455928 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:37.455903 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4783cdd8_1ae0_4520_88b2_c5fc3faf4159.slice/crio-f6986fbd9d3746bba18a629de2498dee93dc78991bd9698b5b0e2bde7662e5a8 WatchSource:0}: Error finding container f6986fbd9d3746bba18a629de2498dee93dc78991bd9698b5b0e2bde7662e5a8: Status 404 returned error can't find the container with id f6986fbd9d3746bba18a629de2498dee93dc78991bd9698b5b0e2bde7662e5a8 Jul 28 17:24:37.471378 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.471354 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5"] Jul 28 17:24:37.474605 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:37.474571 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf135a084_91da_49cb_8a7e_60b9601ab0b4.slice/crio-71d9230c3e0632d07d9c2e01d4d3f4093307b6a785f45d997c5431d58f7fd46a WatchSource:0}: Error finding container 71d9230c3e0632d07d9c2e01d4d3f4093307b6a785f45d997c5431d58f7fd46a: Status 404 returned error can't find the container with id 71d9230c3e0632d07d9c2e01d4d3f4093307b6a785f45d997c5431d58f7fd46a Jul 28 17:24:37.861967 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.861925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" event={"ID":"ae12c5de-a7b3-406a-b792-5cf0d2950ebb","Type":"ContainerStarted","Data":"dda16032d7f9e52946db0fbd3aea6af714f331a621a376f83772845fee671e6c"} Jul 28 17:24:37.862157 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.861997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" event={"ID":"ae12c5de-a7b3-406a-b792-5cf0d2950ebb","Type":"ContainerStarted","Data":"1735406b742481ab66089ca4da30ee4d8bbac9e6ff4d4d45e67c19ce65f2b338"} Jul 28 17:24:37.863145 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.863121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" event={"ID":"f135a084-91da-49cb-8a7e-60b9601ab0b4","Type":"ContainerStarted","Data":"71d9230c3e0632d07d9c2e01d4d3f4093307b6a785f45d997c5431d58f7fd46a"} Jul 28 17:24:37.864518 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.864494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shhb8" event={"ID":"c428648b-bbbc-434c-9d53-2629cdbd4935","Type":"ContainerStarted","Data":"0001106f04ae514241be4fd783341b71b6a17a4badfd622e6536e3658d5a4bd8"} Jul 28 17:24:37.865468 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.865452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc4fb576f-7rttz" Jul 28 17:24:37.865554 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.865471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-m2kzp" event={"ID":"4783cdd8-1ae0-4520-88b2-c5fc3faf4159","Type":"ContainerStarted","Data":"f6986fbd9d3746bba18a629de2498dee93dc78991bd9698b5b0e2bde7662e5a8"} Jul 28 17:24:37.880171 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.880120 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qs6hm" podStartSLOduration=1.880102934 podStartE2EDuration="1.880102934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:37.879049029 +0000 UTC m=+111.031549223" watchObservedRunningTime="2026-07-28 17:24:37.880102934 +0000 UTC m=+111.032603110" Jul 28 17:24:37.907821 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.907783 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cc4fb576f-7rttz"] Jul 28 17:24:37.914518 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:37.914480 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cc4fb576f-7rttz"] Jul 28 17:24:39.377849 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:39.377815 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6fe8ebe-f6bd-4e00-a8bd-cce316794565" path="/var/lib/kubelet/pods/c6fe8ebe-f6bd-4e00-a8bd-cce316794565/volumes" Jul 28 17:24:39.836109 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:39.835712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-25cpf" Jul 28 17:24:39.877307 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:39.877273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" event={"ID":"f135a084-91da-49cb-8a7e-60b9601ab0b4","Type":"ContainerStarted","Data":"1f0e141ad60523f083b376e617a824e6a12e7992459eb96392237cf4a1fedb20"} Jul 28 17:24:39.897795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:39.897527 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zb6h5" podStartSLOduration=3.187595797 podStartE2EDuration="4.897503152s" totalImagesPullingTime="1.709907355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:39.89559291 +0000 UTC m=+113.048093109" watchObservedRunningTime="2026-07-28 17:24:39.897503152 +0000 UTC m=+113.050003327" Jul 28 17:24:41.243596 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.243560 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-f2z88"] Jul 28 17:24:41.249468 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.249008 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.254225 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.254194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 17:24:41.254465 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.254443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 17:24:41.254604 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.254592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xz9pr\"" Jul 28 17:24:41.254745 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.254730 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 17:24:41.261065 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.261040 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-f2z88"] Jul 28 17:24:41.290986 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.290950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51fd1b3f-0da5-4635-aff9-775933b2f13b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.291182 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.290999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbcs\" (UniqueName: \"kubernetes.io/projected/51fd1b3f-0da5-4635-aff9-775933b2f13b-kube-api-access-hrbcs\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.291182 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.291079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51fd1b3f-0da5-4635-aff9-775933b2f13b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.291182 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.291106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51fd1b3f-0da5-4635-aff9-775933b2f13b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.392119 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.392079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51fd1b3f-0da5-4635-aff9-775933b2f13b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.392119 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.392122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51fd1b3f-0da5-4635-aff9-775933b2f13b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.392377 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.392172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51fd1b3f-0da5-4635-aff9-775933b2f13b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.392377 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.392207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbcs\" (UniqueName: \"kubernetes.io/projected/51fd1b3f-0da5-4635-aff9-775933b2f13b-kube-api-access-hrbcs\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.393224 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.393194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51fd1b3f-0da5-4635-aff9-775933b2f13b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.395437 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.395412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51fd1b3f-0da5-4635-aff9-775933b2f13b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.395862 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.395829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51fd1b3f-0da5-4635-aff9-775933b2f13b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.402216 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.402181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbcs\" (UniqueName: \"kubernetes.io/projected/51fd1b3f-0da5-4635-aff9-775933b2f13b-kube-api-access-hrbcs\") pod \"prometheus-operator-84d69f794d-f2z88\" (UID: \"51fd1b3f-0da5-4635-aff9-775933b2f13b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:41.560383 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:41.560306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" Jul 28 17:24:42.020267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:42.020225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-f2z88"] Jul 28 17:24:42.023616 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:42.023586 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fd1b3f_0da5_4635_aff9_775933b2f13b.slice/crio-20fc5fdd84f204d6eae1aceb62aeea6600164efa8ae8a23e490faa13ff15c36e WatchSource:0}: Error finding container 20fc5fdd84f204d6eae1aceb62aeea6600164efa8ae8a23e490faa13ff15c36e: Status 404 returned error can't find the container with id 20fc5fdd84f204d6eae1aceb62aeea6600164efa8ae8a23e490faa13ff15c36e Jul 28 17:24:42.895987 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:42.895886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shhb8" event={"ID":"c428648b-bbbc-434c-9d53-2629cdbd4935","Type":"ContainerStarted","Data":"332cf1ae08047e6602074fdefa6fb72d63a28bbc758c3bc3d1183300c88ac63e"} Jul 28 17:24:42.897857 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:42.897780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" event={"ID":"51fd1b3f-0da5-4635-aff9-775933b2f13b","Type":"ContainerStarted","Data":"20fc5fdd84f204d6eae1aceb62aeea6600164efa8ae8a23e490faa13ff15c36e"} Jul 28 17:24:44.908468 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:44.908434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" event={"ID":"51fd1b3f-0da5-4635-aff9-775933b2f13b","Type":"ContainerStarted","Data":"446e651a9aaa36f545533b948011d6071761893586e150815a511cfc6f01ff48"} Jul 28 17:24:44.908977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:44.908488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" event={"ID":"51fd1b3f-0da5-4635-aff9-775933b2f13b","Type":"ContainerStarted","Data":"446b1e542f47946149f55abca9c26d0de48f99cf0884b01cf18f2c7472140cd9"} Jul 28 17:24:44.933326 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:44.933255 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2z88" podStartSLOduration=1.977517382 podStartE2EDuration="3.933233532s" totalImagesPullingTime="1.95571615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:44.930501598 +0000 UTC m=+118.083001795" watchObservedRunningTime="2026-07-28 17:24:44.933233532 +0000 UTC m=+118.085733709" Jul 28 17:24:45.938406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.938353 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb877976b-r4kg5"] Jul 28 17:24:45.954449 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.954413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb877976b-r4kg5"] Jul 28 17:24:45.954634 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.954531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:45.957806 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.957732 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 17:24:45.959615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.958736 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-frkgx\"" Jul 28 17:24:45.959615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.958795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 17:24:45.959615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.959072 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 17:24:45.959615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.959072 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 17:24:45.959615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.959462 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 17:24:45.966841 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:45.966569 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 17:24:46.034102 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-oauth-serving-cert\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.034282 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfz5l\" (UniqueName: \"kubernetes.io/projected/a3a4be7b-0f58-4c45-a77d-f6f669771a84-kube-api-access-dfz5l\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.034282 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-oauth-config\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.034282 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-config\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.034282 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-trusted-ca-bundle\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.034499 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-service-ca\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.034499 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.034401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-serving-cert\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.135883 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.135840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-oauth-config\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136042 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.135892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-config\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136547 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.136521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-trusted-ca-bundle\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136649 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.136586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-config\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136706 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.136655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-service-ca\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136706 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.136698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-serving-cert\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136847 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.136732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-oauth-serving-cert\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.136847 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.136785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfz5l\" (UniqueName: \"kubernetes.io/projected/a3a4be7b-0f58-4c45-a77d-f6f669771a84-kube-api-access-dfz5l\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.137392 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.137362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-trusted-ca-bundle\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.137719 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.137668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-service-ca\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.138307 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.138284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-oauth-serving-cert\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.139230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.139208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-oauth-config\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.140111 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.140069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-serving-cert\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.146866 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.146813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfz5l\" (UniqueName: \"kubernetes.io/projected/a3a4be7b-0f58-4c45-a77d-f6f669771a84-kube-api-access-dfz5l\") pod \"console-cb877976b-r4kg5\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.268165 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.268131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:24:46.432563 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.432513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb877976b-r4kg5"] Jul 28 17:24:46.435898 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:46.435865 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a4be7b_0f58_4c45_a77d_f6f669771a84.slice/crio-a87679a2742a54be334cef0683acc933860c6e41d466a26736b80a134bfa55ec WatchSource:0}: Error finding container a87679a2742a54be334cef0683acc933860c6e41d466a26736b80a134bfa55ec: Status 404 returned error can't find the container with id a87679a2742a54be334cef0683acc933860c6e41d466a26736b80a134bfa55ec Jul 28 17:24:46.591554 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.591512 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw"] Jul 28 17:24:46.594319 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.594294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.597288 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.597215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qfzcg\"" Jul 28 17:24:46.597288 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.597215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:24:46.597511 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.597217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 17:24:46.616845 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.616802 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw"] Jul 28 17:24:46.617852 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.617818 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb"] Jul 28 17:24:46.620593 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.620569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.623888 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.623862 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:24:46.624196 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.624177 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 17:24:46.624344 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.624292 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 17:24:46.624409 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.624351 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-64m2c\"" Jul 28 17:24:46.629810 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.629785 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cbmtz"] Jul 28 17:24:46.632428 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.632406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.636177 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.636147 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 17:24:46.636543 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.636527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n84sh\"" Jul 28 17:24:46.636672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.636651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb"] Jul 28 17:24:46.636825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.636809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 17:24:46.636893 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.636853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 17:24:46.641454 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dce10a2-a57e-4da6-a1da-e98730fa9897-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.641587 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.641587 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.641587 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.641587 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6hk\" (UniqueName: \"kubernetes.io/projected/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-api-access-lm6hk\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.641807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.641807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.641807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.641807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j275\" (UniqueName: \"kubernetes.io/projected/5dce10a2-a57e-4da6-a1da-e98730fa9897-kube-api-access-7j275\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.641807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.641800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.742869 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.742830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-tls\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.742878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.742919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.743064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.742951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j275\" (UniqueName: \"kubernetes.io/projected/5dce10a2-a57e-4da6-a1da-e98730fa9897-kube-api-access-7j275\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.743064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.742975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-root\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.743064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nsn\" (UniqueName: \"kubernetes.io/projected/7006fefc-7d0d-4b0a-a026-83a18e65014b-kube-api-access-42nsn\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-wtmp\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dce10a2-a57e-4da6-a1da-e98730fa9897-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.743345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.743345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.743345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.743345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-textfile\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743618 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7006fefc-7d0d-4b0a-a026-83a18e65014b-metrics-client-ca\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743618 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.743618 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6hk\" (UniqueName: \"kubernetes.io/projected/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-api-access-lm6hk\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.743618 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.743618 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.743874 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:46.743629 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 17:24:46.743952 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.743925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dce10a2-a57e-4da6-a1da-e98730fa9897-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.744151 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.744125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.744230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.744124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.744294 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:46.744229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-tls podName:5dce10a2-a57e-4da6-a1da-e98730fa9897 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:47.243773476 +0000 UTC m=+120.396273644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-nwgmw" (UID: "5dce10a2-a57e-4da6-a1da-e98730fa9897") : secret "openshift-state-metrics-tls" not found Jul 28 17:24:46.744357 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.744291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-sys\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.744408 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.744380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.746021 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.745989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.746335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.746304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.746545 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.746525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.759543 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.759512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6hk\" (UniqueName: \"kubernetes.io/projected/3ff56a99-8ba5-4a79-ba71-1a7e393d64f4-kube-api-access-lm6hk\") pod \"kube-state-metrics-56d9f944b4-kxrdb\" (UID: \"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.763570 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.763540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j275\" (UniqueName: \"kubernetes.io/projected/5dce10a2-a57e-4da6-a1da-e98730fa9897-kube-api-access-7j275\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:46.845513 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-tls\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845513 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845513 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-root\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nsn\" (UniqueName: \"kubernetes.io/projected/7006fefc-7d0d-4b0a-a026-83a18e65014b-kube-api-access-42nsn\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-wtmp\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-textfile\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7006fefc-7d0d-4b0a-a026-83a18e65014b-metrics-client-ca\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.845783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-sys\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.846571 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.845821 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-sys\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.846571 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:46.845927 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 17:24:46.846571 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:24:46.845991 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-tls podName:7006fefc-7d0d-4b0a-a026-83a18e65014b nodeName:}" failed. No retries permitted until 2026-07-28 17:24:47.345971123 +0000 UTC m=+120.498471277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-tls") pod "node-exporter-cbmtz" (UID: "7006fefc-7d0d-4b0a-a026-83a18e65014b") : secret "node-exporter-tls" not found Jul 28 17:24:46.846571 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.846043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-root\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.846724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.846616 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7006fefc-7d0d-4b0a-a026-83a18e65014b-metrics-client-ca\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.846724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.846685 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-wtmp\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.846986 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.846957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-textfile\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.847128 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.847105 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.849211 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.849188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.858345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.858300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nsn\" (UniqueName: \"kubernetes.io/projected/7006fefc-7d0d-4b0a-a026-83a18e65014b-kube-api-access-42nsn\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:46.924024 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.923985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shhb8" event={"ID":"c428648b-bbbc-434c-9d53-2629cdbd4935","Type":"ContainerStarted","Data":"fbed8fa8f2e4d82ecd6e981e4c564718d3e64f86df3b5e8ad802b38a7cfdd4c9"} Jul 28 17:24:46.925185 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.925161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb877976b-r4kg5" event={"ID":"a3a4be7b-0f58-4c45-a77d-f6f669771a84","Type":"ContainerStarted","Data":"a87679a2742a54be334cef0683acc933860c6e41d466a26736b80a134bfa55ec"} Jul 28 17:24:46.933121 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.933092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" Jul 28 17:24:46.945386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:46.945339 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-shhb8" podStartSLOduration=33.218824953 podStartE2EDuration="46.945323225s" totalImagesPullingTime="13.726498272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:46.944663597 +0000 UTC m=+120.097163798" watchObservedRunningTime="2026-07-28 17:24:46.945323225 +0000 UTC m=+120.097823401" Jul 28 17:24:47.082797 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.082738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb"] Jul 28 17:24:47.085955 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:47.085926 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff56a99_8ba5_4a79_ba71_1a7e393d64f4.slice/crio-994b75b021e3c8b6594b391e209b6fdc2f3a365c543394825a4043c327702e94 WatchSource:0}: Error finding container 994b75b021e3c8b6594b391e209b6fdc2f3a365c543394825a4043c327702e94: Status 404 returned error can't find the container with id 994b75b021e3c8b6594b391e209b6fdc2f3a365c543394825a4043c327702e94 Jul 28 17:24:47.249350 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.249252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:47.252603 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.252540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dce10a2-a57e-4da6-a1da-e98730fa9897-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nwgmw\" (UID: \"5dce10a2-a57e-4da6-a1da-e98730fa9897\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:47.350675 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.350635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-tls\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:47.354527 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.354455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7006fefc-7d0d-4b0a-a026-83a18e65014b-node-exporter-tls\") pod \"node-exporter-cbmtz\" (UID: \"7006fefc-7d0d-4b0a-a026-83a18e65014b\") " pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:47.512021 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.511931 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qfzcg\"" Jul 28 17:24:47.517109 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.517072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" Jul 28 17:24:47.549034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.549002 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n84sh\"" Jul 28 17:24:47.555847 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.555618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbmtz" Jul 28 17:24:47.572032 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:47.571920 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7006fefc_7d0d_4b0a_a026_83a18e65014b.slice/crio-ce023b5290f1e75a9702d3a85d3eff8e4996c9cdf5808b35410680be85fea474 WatchSource:0}: Error finding container ce023b5290f1e75a9702d3a85d3eff8e4996c9cdf5808b35410680be85fea474: Status 404 returned error can't find the container with id ce023b5290f1e75a9702d3a85d3eff8e4996c9cdf5808b35410680be85fea474 Jul 28 17:24:47.714461 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.714411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw"] Jul 28 17:24:47.719741 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:47.719708 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dce10a2_a57e_4da6_a1da_e98730fa9897.slice/crio-0d38fe6ebf0b573e68b0c9833e7735fb11dd4a46dbf9de52a1a9e2840ed1ed6c WatchSource:0}: Error finding container 0d38fe6ebf0b573e68b0c9833e7735fb11dd4a46dbf9de52a1a9e2840ed1ed6c: Status 404 returned error can't find the container with id 0d38fe6ebf0b573e68b0c9833e7735fb11dd4a46dbf9de52a1a9e2840ed1ed6c Jul 28 17:24:47.933409 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.933291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmtz" event={"ID":"7006fefc-7d0d-4b0a-a026-83a18e65014b","Type":"ContainerStarted","Data":"ce023b5290f1e75a9702d3a85d3eff8e4996c9cdf5808b35410680be85fea474"} Jul 28 17:24:47.939683 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.939614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" event={"ID":"5dce10a2-a57e-4da6-a1da-e98730fa9897","Type":"ContainerStarted","Data":"f49357627529f338781b5e8581fdab2a806868f38ebc65919cabbd370af137d7"} Jul 28 17:24:47.939683 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.939688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" event={"ID":"5dce10a2-a57e-4da6-a1da-e98730fa9897","Type":"ContainerStarted","Data":"bf5fce28c40e5ce47279ff1f0952add0ba603d303432338e829eb966c0a53a9b"} Jul 28 17:24:47.939924 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.939718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" event={"ID":"5dce10a2-a57e-4da6-a1da-e98730fa9897","Type":"ContainerStarted","Data":"0d38fe6ebf0b573e68b0c9833e7735fb11dd4a46dbf9de52a1a9e2840ed1ed6c"} Jul 28 17:24:47.942524 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:47.942444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" event={"ID":"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4","Type":"ContainerStarted","Data":"994b75b021e3c8b6594b391e209b6fdc2f3a365c543394825a4043c327702e94"} Jul 28 17:24:49.730303 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.730266 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-cdb8d69-rlllz"] Jul 28 17:24:49.734195 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.734165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.737389 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.737362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 17:24:49.737533 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.737361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8v8fqdn4blmue\"" Jul 28 17:24:49.737533 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.737361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 17:24:49.737630 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.737364 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bk6pg\"" Jul 28 17:24:49.737630 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.737528 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 17:24:49.738749 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.738721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 17:24:49.738891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.738722 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 17:24:49.748057 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.748028 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-cdb8d69-rlllz"] Jul 28 17:24:49.777629 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.777833 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.777833 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-tls\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.777833 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-grpc-tls\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.777961 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-metrics-client-ca\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.777999 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.778043 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.777999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5s2\" (UniqueName: \"kubernetes.io/projected/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-kube-api-access-lf5s2\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.778043 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.778033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878747 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-metrics-client-ca\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5s2\" (UniqueName: \"kubernetes.io/projected/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-kube-api-access-lf5s2\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.878945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-tls\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.879207 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.878963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-grpc-tls\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.879608 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.879559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-metrics-client-ca\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.882159 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.882134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-tls\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.882281 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.882134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.882341 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.882285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.882452 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.882424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-grpc-tls\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.882788 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.882741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.883088 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.883067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:49.887870 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:49.887838 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5s2\" (UniqueName: \"kubernetes.io/projected/0fefb1e4-4bbc-482c-9b23-40390e1a66c1-kube-api-access-lf5s2\") pod \"thanos-querier-cdb8d69-rlllz\" (UID: \"0fefb1e4-4bbc-482c-9b23-40390e1a66c1\") " pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:50.046219 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:50.046122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:24:51.404243 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.404203 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql"] Jul 28 17:24:51.407930 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.407901 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.411042 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.411009 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 17:24:51.411283 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.411249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hw4s5\"" Jul 28 17:24:51.420792 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.420721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql"] Jul 28 17:24:51.493300 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.493264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4zb\" (UniqueName: \"kubernetes.io/projected/ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7-kube-api-access-9d4zb\") pod \"monitoring-plugin-68b7fc9574-m9dql\" (UID: \"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.493494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.493349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m9dql\" (UID: \"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.594355 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.594319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m9dql\" (UID: \"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.594516 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.594440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4zb\" (UniqueName: \"kubernetes.io/projected/ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7-kube-api-access-9d4zb\") pod \"monitoring-plugin-68b7fc9574-m9dql\" (UID: \"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.597178 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.597144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m9dql\" (UID: \"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.606284 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.606251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4zb\" (UniqueName: \"kubernetes.io/projected/ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7-kube-api-access-9d4zb\") pod \"monitoring-plugin-68b7fc9574-m9dql\" (UID: \"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:51.719443 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:51.719340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:24:52.967812 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.967768 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:24:52.974180 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.974124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:52.977793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.977737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:24:52.978036 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.977743 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:24:52.979421 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.979394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:24:52.979874 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.979681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:24:52.980688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.980236 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:24:52.981091 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.981063 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 17:24:52.982304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.981529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:24:52.982304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.981741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:24:52.982783 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.982691 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:24:52.983840 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.983408 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:24:52.983840 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.983476 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-195u0v0am7fob\"" Jul 28 17:24:52.983840 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.983637 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:24:52.983840 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.983666 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wf6rm\"" Jul 28 17:24:52.983840 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.983718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:24:52.984132 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.984036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:24:52.984132 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.984068 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:24:52.987410 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.987383 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:24:52.996972 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:52.996867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:24:53.117559 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117559 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bhh\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-kube-api-access-s2bhh\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.117831 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.117982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.118030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.118062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.118093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.118119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.118144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.118269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.118171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.218911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.218954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.218977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.218995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bhh\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-kube-api-access-s2bhh\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.219656 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.220409 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.219679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.220544 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.220515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.221068 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.220742 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.222081 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.222044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.226153 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.226117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.226487 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.226459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.226947 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.226925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.227312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.227290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.227673 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.227651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.227912 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.227675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.228025 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.227823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.228137 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.228053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.228396 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.228372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.229202 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.229161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.229575 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.229514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.230485 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.230415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.230597 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.230501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.230818 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.230747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.231058 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.230929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.238697 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.238629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bhh\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-kube-api-access-s2bhh\") pod \"prometheus-k8s-0\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:53.291092 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:53.291052 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:24:54.169694 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:54.169659 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb877976b-r4kg5"] Jul 28 17:24:57.162876 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:57.162834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:24:57.165492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:57.165461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669bb6ec-59b7-47cc-b345-43db24f5c868-metrics-certs\") pod \"network-metrics-daemon-n8rn6\" (UID: \"669bb6ec-59b7-47cc-b345-43db24f5c868\") " pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:24:57.184938 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:57.184906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpv4x\"" Jul 28 17:24:57.192795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:57.192738 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8rn6" Jul 28 17:24:58.353768 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:58.353715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:24:58.398447 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:58.386205 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql"] Jul 28 17:24:58.398447 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:58.395816 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceff7ea7_5a00_4ed1_bbc6_94633cefaaa7.slice/crio-7744fc657e86aa332da9b7ffe8a85535d43fcaa1c137fddbdec34781e9437703 WatchSource:0}: Error finding container 7744fc657e86aa332da9b7ffe8a85535d43fcaa1c137fddbdec34781e9437703: Status 404 returned error can't find the container with id 7744fc657e86aa332da9b7ffe8a85535d43fcaa1c137fddbdec34781e9437703 Jul 28 17:24:58.424251 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:58.424188 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n8rn6"] Jul 28 17:24:58.448269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:58.447432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-cdb8d69-rlllz"] Jul 28 17:24:58.450657 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:24:58.450616 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fefb1e4_4bbc_482c_9b23_40390e1a66c1.slice/crio-ac9432112c38afe2d3df5a149939b05fef685354878c4649e06cc2b06a512dcc WatchSource:0}: Error finding container ac9432112c38afe2d3df5a149939b05fef685354878c4649e06cc2b06a512dcc: Status 404 returned error can't find the container with id ac9432112c38afe2d3df5a149939b05fef685354878c4649e06cc2b06a512dcc Jul 28 17:24:58.998002 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:58.997960 2564 generic.go:356] "Generic (PLEG): container finished" podID="7006fefc-7d0d-4b0a-a026-83a18e65014b" containerID="bc58a61ccc0f3d092279cc7da6537ef010a4ce64492e47a7fee459cbb7c576cd" exitCode=0 Jul 28 17:24:58.999262 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:58.999039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmtz" event={"ID":"7006fefc-7d0d-4b0a-a026-83a18e65014b","Type":"ContainerDied","Data":"bc58a61ccc0f3d092279cc7da6537ef010a4ce64492e47a7fee459cbb7c576cd"} Jul 28 17:24:59.003329 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.003293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb877976b-r4kg5" event={"ID":"a3a4be7b-0f58-4c45-a77d-f6f669771a84","Type":"ContainerStarted","Data":"82be63a10270e0a1c5065956992659a4d29d50d82053ab513eb6158de010b57d"} Jul 28 17:24:59.014168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.014124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" event={"ID":"5dce10a2-a57e-4da6-a1da-e98730fa9897","Type":"ContainerStarted","Data":"e9367f541af25822bcb92b86b00b1da63f85d3968bb2a08a301e5ce092f47eba"} Jul 28 17:24:59.017339 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.017304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"ac9432112c38afe2d3df5a149939b05fef685354878c4649e06cc2b06a512dcc"} Jul 28 17:24:59.020468 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.019844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-m2kzp" event={"ID":"4783cdd8-1ae0-4520-88b2-c5fc3faf4159","Type":"ContainerStarted","Data":"6ad17aa655f222bbeaaa4475afdd6ff7c8b200ecdbeab85aafcd8578446cd2f3"} Jul 28 17:24:59.020468 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.019907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:59.023076 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.022963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"774d935c30b58156397c7f0492afdfe206a0119c305e86183215fce7bccbdab2"} Jul 28 17:24:59.028974 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.028885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" event={"ID":"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4","Type":"ContainerStarted","Data":"1a449bb3c9526241b0a24294846112b4011ae53e6a2d3d67acb9d572126f24c2"} Jul 28 17:24:59.028974 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.028961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" event={"ID":"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4","Type":"ContainerStarted","Data":"547fcd0bb89bd01f1ae03936d5e4a9c2fdf538a0e59150558fa8da782a3c10e0"} Jul 28 17:24:59.028974 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.028989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" event={"ID":"3ff56a99-8ba5-4a79-ba71-1a7e393d64f4","Type":"ContainerStarted","Data":"4e3d9a7c63003c8366069f9041cd2f5de2ab7c68e09d14a5b8dc5f4214245bba"} Jul 28 17:24:59.030045 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.029614 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-m2kzp" Jul 28 17:24:59.033859 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.033345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" event={"ID":"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7","Type":"ContainerStarted","Data":"7744fc657e86aa332da9b7ffe8a85535d43fcaa1c137fddbdec34781e9437703"} Jul 28 17:24:59.036941 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.036481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8rn6" event={"ID":"669bb6ec-59b7-47cc-b345-43db24f5c868","Type":"ContainerStarted","Data":"7ca377e7321b36a8352a5e6f4cbf7dd2c33091a9edfc70be50191f268402ecf5"} Jul 28 17:24:59.067146 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.066658 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-m2kzp" podStartSLOduration=2.334049753 podStartE2EDuration="23.066636747s" totalImagesPullingTime="20.732586994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:59.065372763 +0000 UTC m=+132.217872957" watchObservedRunningTime="2026-07-28 17:24:59.066636747 +0000 UTC m=+132.219136924" Jul 28 17:24:59.094437 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.094381 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nwgmw" podStartSLOduration=2.8148267860000002 podStartE2EDuration="13.094360179s" totalImagesPullingTime="10.279533393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:59.093317697 +0000 UTC m=+132.245817894" watchObservedRunningTime="2026-07-28 17:24:59.094360179 +0000 UTC m=+132.246860356" Jul 28 17:24:59.119053 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.117074 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cb877976b-r4kg5" podStartSLOduration=2.453998726 podStartE2EDuration="14.117052784s" totalImagesPullingTime="11.663054058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:59.116517363 +0000 UTC m=+132.269017538" watchObservedRunningTime="2026-07-28 17:24:59.117052784 +0000 UTC m=+132.269552963" Jul 28 17:24:59.142846 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:24:59.141556 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kxrdb" podStartSLOduration=2.141411214 podStartE2EDuration="13.141505478s" totalImagesPullingTime="11.000094264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:59.140373653 +0000 UTC m=+132.292873849" watchObservedRunningTime="2026-07-28 17:24:59.141505478 +0000 UTC m=+132.294005658" Jul 28 17:25:00.065982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:00.064068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmtz" event={"ID":"7006fefc-7d0d-4b0a-a026-83a18e65014b","Type":"ContainerStarted","Data":"8c7b2daa7fc6d71a34dcb49ed81f82e4747bbe265ae54acb519576484020cfce"} Jul 28 17:25:00.065982 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:00.064178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmtz" event={"ID":"7006fefc-7d0d-4b0a-a026-83a18e65014b","Type":"ContainerStarted","Data":"c8da4dbab9a36da265be51c6d577ef2e3e50e847509e6288ad641aa34690b8d8"} Jul 28 17:25:00.091869 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:00.091219 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cbmtz" podStartSLOduration=3.409800835 podStartE2EDuration="14.091197087s" totalImagesPullingTime="10.522332278s" totalInitContainerRuntime="159.063974ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:00.088436963 +0000 UTC m=+133.240937161" watchObservedRunningTime="2026-07-28 17:25:00.091197087 +0000 UTC m=+133.243697264" Jul 28 17:25:04.092884 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.092685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"bdd9475692cb43f2839b6ee7abb42f1fbc297cd5e0d556d2bb698c948d9f936a"} Jul 28 17:25:04.092884 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.092793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"519cdb9139df1a49c866fad8f217995036fb8c1c88b5b1f315f5534335ea8044"} Jul 28 17:25:04.092884 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.092835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"0467b795227d01b7e1314fb2c86a85105ca6cc3e003126b3af9a8116dc3aa5d4"} Jul 28 17:25:04.096906 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.096865 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" exitCode=0 Jul 28 17:25:04.097080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.096992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} Jul 28 17:25:04.103937 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.103797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" event={"ID":"ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7","Type":"ContainerStarted","Data":"b3443776901075f106c9e2ad251f2190f6247b4da873d81280d2bde5de83bc10"} Jul 28 17:25:04.104318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.104122 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:25:04.111393 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.110808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8rn6" event={"ID":"669bb6ec-59b7-47cc-b345-43db24f5c868","Type":"ContainerStarted","Data":"81281577ea413574a40f1517c5525668d072afe802285ae64c9f53422b32e425"} Jul 28 17:25:04.111393 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.111164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8rn6" event={"ID":"669bb6ec-59b7-47cc-b345-43db24f5c868","Type":"ContainerStarted","Data":"d4f9c7a756a73a7fba47cce8e4b8f5bda6ae35a02b8dc0341a3d282f279ac41e"} Jul 28 17:25:04.115627 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.115595 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" Jul 28 17:25:04.222566 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.222500 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n8rn6" podStartSLOduration=132.691141005 podStartE2EDuration="2m17.222476601s" totalImagesPullingTime="4.531335596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:04.220697378 +0000 UTC m=+137.373197603" watchObservedRunningTime="2026-07-28 17:25:04.222476601 +0000 UTC m=+137.374976778" Jul 28 17:25:04.250160 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:04.250052 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m9dql" podStartSLOduration=8.699491307 podStartE2EDuration="13.250030303s" totalImagesPullingTime="4.550538996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:04.248851475 +0000 UTC m=+137.401351669" watchObservedRunningTime="2026-07-28 17:25:04.250030303 +0000 UTC m=+137.402530480" Jul 28 17:25:05.124827 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:05.124751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"1214743eb36e76ba7dc0694856bf5f10badc8b4f7b61c871621a36356360ce32"} Jul 28 17:25:06.134016 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:06.133886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"6a8fb7729b6bed1ac1b5ac50e17c4e41cc365be76e73e0c353e08048fd63d7e1"} Jul 28 17:25:06.135431 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:06.134329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" event={"ID":"0fefb1e4-4bbc-482c-9b23-40390e1a66c1","Type":"ContainerStarted","Data":"531447da558381eb84bc13837c207570184b2233320fdef726d61462e04e78bd"} Jul 28 17:25:06.135431 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:06.134386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:25:06.160654 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:06.160584 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" podStartSLOduration=11.204195769 podStartE2EDuration="17.160560381s" totalImagesPullingTime="5.956364612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:06.157928395 +0000 UTC m=+139.310428563" watchObservedRunningTime="2026-07-28 17:25:06.160560381 +0000 UTC m=+139.313060564" Jul 28 17:25:06.269296 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:06.269259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:25:09.150385 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.150352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} Jul 28 17:25:09.150751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.150446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} Jul 28 17:25:09.150751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.150489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} Jul 28 17:25:09.150751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.150519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} Jul 28 17:25:09.150751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.150541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} Jul 28 17:25:09.150751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.150581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerStarted","Data":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} Jul 28 17:25:09.193583 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:09.193509 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.458175453 podStartE2EDuration="17.193486636s" totalImagesPullingTime="8.715502677s" totalInitContainerRuntime="19.808506ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:09.191304909 +0000 UTC m=+142.343805138" watchObservedRunningTime="2026-07-28 17:25:09.193486636 +0000 UTC m=+142.345986814" Jul 28 17:25:12.145484 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:12.145454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-cdb8d69-rlllz" Jul 28 17:25:13.291969 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:13.291935 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:25:16.181784 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:16.181719 2564 generic.go:356] "Generic (PLEG): container finished" podID="be13f61c-c224-4f9e-b233-f8b9b2d16968" containerID="1941e94c097ea0ca3b1deb50df7ed552a0501b25a9fc8733cf60ce73cae7b329" exitCode=0 Jul 28 17:25:16.182184 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:16.181794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" event={"ID":"be13f61c-c224-4f9e-b233-f8b9b2d16968","Type":"ContainerDied","Data":"1941e94c097ea0ca3b1deb50df7ed552a0501b25a9fc8733cf60ce73cae7b329"} Jul 28 17:25:16.182184 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:16.182155 2564 scope.go:117] "RemoveContainer" containerID="1941e94c097ea0ca3b1deb50df7ed552a0501b25a9fc8733cf60ce73cae7b329" Jul 28 17:25:17.188543 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:17.188510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fggn6" event={"ID":"be13f61c-c224-4f9e-b233-f8b9b2d16968","Type":"ContainerStarted","Data":"3b7dfef3102020262b3122e5560c6bbb76ac1cfc60f26ed237a8be48e44ccf6c"} Jul 28 17:25:20.206149 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:20.206065 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b0bb629-7283-4f68-8e90-c392c834a0e5" containerID="2da91ef2cd372e7918f1bd02d53a0d6b1b6e0f0d3447396094bb0fee9e77b25d" exitCode=0 Jul 28 17:25:20.206610 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:20.206139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-msvxc" event={"ID":"5b0bb629-7283-4f68-8e90-c392c834a0e5","Type":"ContainerDied","Data":"2da91ef2cd372e7918f1bd02d53a0d6b1b6e0f0d3447396094bb0fee9e77b25d"} Jul 28 17:25:20.206610 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:20.206488 2564 scope.go:117] "RemoveContainer" containerID="2da91ef2cd372e7918f1bd02d53a0d6b1b6e0f0d3447396094bb0fee9e77b25d" Jul 28 17:25:21.212207 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:21.212172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-msvxc" event={"ID":"5b0bb629-7283-4f68-8e90-c392c834a0e5","Type":"ContainerStarted","Data":"e31fa6aafe5062ec4afbe5e55f2177de3270bd99b994f29cd3ddc24637e4cab5"} Jul 28 17:25:24.030872 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.030821 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cb877976b-r4kg5" podUID="a3a4be7b-0f58-4c45-a77d-f6f669771a84" containerName="console" containerID="cri-o://82be63a10270e0a1c5065956992659a4d29d50d82053ab513eb6158de010b57d" gracePeriod=15 Jul 28 17:25:24.225682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.225650 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3a4be7b-0f58-4c45-a77d-f6f669771a84" containerID="82be63a10270e0a1c5065956992659a4d29d50d82053ab513eb6158de010b57d" exitCode=0 Jul 28 17:25:24.225875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.225716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb877976b-r4kg5" event={"ID":"a3a4be7b-0f58-4c45-a77d-f6f669771a84","Type":"ContainerDied","Data":"82be63a10270e0a1c5065956992659a4d29d50d82053ab513eb6158de010b57d"} Jul 28 17:25:24.313039 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.313007 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:25:24.424464 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424420 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfz5l\" (UniqueName: \"kubernetes.io/projected/a3a4be7b-0f58-4c45-a77d-f6f669771a84-kube-api-access-dfz5l\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424464 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-service-ca\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-trusted-ca-bundle\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424542 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-oauth-config\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424575 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-serving-cert\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-oauth-serving-cert\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424653 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-config\") pod \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\" (UID: \"a3a4be7b-0f58-4c45-a77d-f6f669771a84\") " Jul 28 17:25:24.424983 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.424955 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-service-ca" (OuterVolumeSpecName: "service-ca") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:25:24.425050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.425023 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:25:24.425050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.425031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:25:24.425142 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.425094 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-config" (OuterVolumeSpecName: "console-config") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:25:24.425142 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.425101 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-service-ca\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:24.426815 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.426792 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a4be7b-0f58-4c45-a77d-f6f669771a84-kube-api-access-dfz5l" (OuterVolumeSpecName: "kube-api-access-dfz5l") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "kube-api-access-dfz5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:25:24.426815 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.426791 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:25:24.426945 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.426801 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a3a4be7b-0f58-4c45-a77d-f6f669771a84" (UID: "a3a4be7b-0f58-4c45-a77d-f6f669771a84"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:25:24.525688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.525645 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfz5l\" (UniqueName: \"kubernetes.io/projected/a3a4be7b-0f58-4c45-a77d-f6f669771a84-kube-api-access-dfz5l\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:24.525688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.525679 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-trusted-ca-bundle\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:24.525688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.525689 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-oauth-config\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:24.525688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.525697 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-serving-cert\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:24.525967 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.525705 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-oauth-serving-cert\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:24.525967 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:24.525714 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3a4be7b-0f58-4c45-a77d-f6f669771a84-console-config\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:25:25.231639 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:25.231602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb877976b-r4kg5" event={"ID":"a3a4be7b-0f58-4c45-a77d-f6f669771a84","Type":"ContainerDied","Data":"a87679a2742a54be334cef0683acc933860c6e41d466a26736b80a134bfa55ec"} Jul 28 17:25:25.231639 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:25.231640 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb877976b-r4kg5" Jul 28 17:25:25.232119 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:25.231674 2564 scope.go:117] "RemoveContainer" containerID="82be63a10270e0a1c5065956992659a4d29d50d82053ab513eb6158de010b57d" Jul 28 17:25:25.254352 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:25.254319 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb877976b-r4kg5"] Jul 28 17:25:25.257183 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:25.257157 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cb877976b-r4kg5"] Jul 28 17:25:25.375975 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:25.375942 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a4be7b-0f58-4c45-a77d-f6f669771a84" path="/var/lib/kubelet/pods/a3a4be7b-0f58-4c45-a77d-f6f669771a84/volumes" Jul 28 17:25:53.292429 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:53.292387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:25:53.311535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:53.311505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:25:53.379064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:25:53.379033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:11.437196 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.437159 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:26:11.438168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.438130 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="prometheus" containerID="cri-o://ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" gracePeriod=600 Jul 28 17:26:11.438973 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.438348 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" gracePeriod=600 Jul 28 17:26:11.439084 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.438618 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-web" containerID="cri-o://85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" gracePeriod=600 Jul 28 17:26:11.439084 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.438639 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy" containerID="cri-o://3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" gracePeriod=600 Jul 28 17:26:11.439191 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.438658 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="thanos-sidecar" containerID="cri-o://f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" gracePeriod=600 Jul 28 17:26:11.439243 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.438672 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="config-reloader" containerID="cri-o://735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" gracePeriod=600 Jul 28 17:26:11.710785 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.710735 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-thanos-prometheus-http-client-file\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720100 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-kube-rbac-proxy\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720131 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-trusted-ca-bundle\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720184 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config-out\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-0\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720235 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-web-config\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720259 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-2\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720281 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-db\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720315 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-metrics-client-ca\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2bhh\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-kube-api-access-s2bhh\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720367 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-1\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720395 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-grpc-tls\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720471 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-kubelet-serving-ca-bundle\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720497 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-tls-assets\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720539 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-serving-certs-ca-bundle\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.721241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.722316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-metrics-client-certs\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.722316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.720625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-tls\") pod \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\" (UID: \"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1\") " Jul 28 17:26:11.722316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.721487 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.722316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.721885 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.722316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.721996 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.722862 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.722572 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.722862 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.722799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.723044 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.722911 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:26:11.723832 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.723799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.723954 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.723932 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config-out" (OuterVolumeSpecName: "config-out") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:26:11.724696 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.724618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.725922 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.725890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-kube-api-access-s2bhh" (OuterVolumeSpecName: "kube-api-access-s2bhh") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "kube-api-access-s2bhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:26:11.726050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.725919 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.726050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.726012 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.726205 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.726182 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.726457 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.726431 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:26:11.726530 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.726471 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config" (OuterVolumeSpecName: "config") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.726649 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.726623 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:26:11.726794 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.726750 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.728208 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.728175 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.728391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.728373 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.739864 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.739825 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-web-config" (OuterVolumeSpecName: "web-config") pod "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" (UID: "ca62eb39-29c0-4ca3-99ed-f09e2a6013a1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:26:11.821853 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821810 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.821853 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821849 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.821853 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821862 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config-out\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821870 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821902 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-web-config\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821911 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821921 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-db\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821930 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-metrics-client-ca\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821938 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bhh\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-kube-api-access-s2bhh\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821946 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821954 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-grpc-tls\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821962 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-config\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821969 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821977 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-tls-assets\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821985 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.821997 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.822006 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-metrics-client-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.822015 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.822023 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:11.822090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:11.822031 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1-secret-kube-rbac-proxy\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:26:12.441129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441096 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" exitCode=0 Jul 28 17:26:12.441129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441121 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" exitCode=0 Jul 28 17:26:12.441129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441127 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" exitCode=0 Jul 28 17:26:12.441129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441133 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" exitCode=0 Jul 28 17:26:12.441129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441138 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" exitCode=0 Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441143 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" exitCode=0 Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441177 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441328 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} Jul 28 17:26:12.441722 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.441369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca62eb39-29c0-4ca3-99ed-f09e2a6013a1","Type":"ContainerDied","Data":"774d935c30b58156397c7f0492afdfe206a0119c305e86183215fce7bccbdab2"} Jul 28 17:26:12.450020 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.449993 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.468662 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.468628 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:26:12.472636 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.472605 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:26:12.474559 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.474543 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.482298 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.482271 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.489821 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.489799 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.497744 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.497721 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.506178 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506142 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:26:12.506568 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506550 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy" Jul 28 17:26:12.506568 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506569 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506581 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-thanos" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506588 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-thanos" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506598 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="prometheus" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506603 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="prometheus" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506610 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="thanos-sidecar" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506634 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="thanos-sidecar" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506645 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="config-reloader" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506651 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="config-reloader" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506661 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-web" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506667 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-web" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506675 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="init-config-reloader" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506680 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="init-config-reloader" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506693 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a4be7b-0f58-4c45-a77d-f6f669771a84" containerName="console" Jul 28 17:26:12.506701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506698 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a4be7b-0f58-4c45-a77d-f6f669771a84" containerName="console" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506774 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="prometheus" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506786 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506796 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a4be7b-0f58-4c45-a77d-f6f669771a84" containerName="console" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506806 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="thanos-sidecar" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506813 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-web" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506821 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="kube-rbac-proxy-thanos" Jul 28 17:26:12.507335 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.506830 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" containerName="config-reloader" Jul 28 17:26:12.510775 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.510738 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.516702 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.515749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.519933 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.519865 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wf6rm\"" Jul 28 17:26:12.519933 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.519928 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:26:12.520171 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.519945 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:26:12.520171 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.519958 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:26:12.520171 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.519973 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:26:12.520280 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520246 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-195u0v0am7fob\"" Jul 28 17:26:12.520377 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520351 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:26:12.520453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520394 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:26:12.520507 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520457 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:26:12.520557 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520514 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:26:12.520557 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520542 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:26:12.520650 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520572 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 17:26:12.520864 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:26:12.520944 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520894 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:26:12.520944 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.520924 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:26:12.522222 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.522182 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.522603 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.522582 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.522682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.522610 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} err="failed to get container status \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" Jul 28 17:26:12.522682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.522640 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.523142 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.522966 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.523142 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.522997 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} err="failed to get container status \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" Jul 28 17:26:12.523142 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.523021 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.523417 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.523343 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.523417 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.523376 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} err="failed to get container status \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" Jul 28 17:26:12.523417 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.523397 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.523696 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.523667 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.523776 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.523700 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} err="failed to get container status \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" Jul 28 17:26:12.523776 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.523714 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.524005 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.523981 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.524064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524011 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} err="failed to get container status \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" Jul 28 17:26:12.524064 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524031 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.524366 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.524276 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.524366 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524313 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} err="failed to get container status \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" Jul 28 17:26:12.524366 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524328 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.524809 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:26:12.524599 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.524809 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524628 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} err="failed to get container status \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" Jul 28 17:26:12.524809 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524649 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.525034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.524990 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} err="failed to get container status \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" Jul 28 17:26:12.525034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525013 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.525312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525279 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} err="failed to get container status \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" Jul 28 17:26:12.525312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525307 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.525630 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525592 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} err="failed to get container status \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" Jul 28 17:26:12.525630 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525615 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.525904 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525885 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:26:12.525986 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525936 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} err="failed to get container status \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" Jul 28 17:26:12.525986 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.525958 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.526236 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.526214 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} err="failed to get container status \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" Jul 28 17:26:12.526236 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.526234 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.526494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.526476 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} err="failed to get container status \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" Jul 28 17:26:12.526494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.526494 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.526733 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.526714 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} err="failed to get container status \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" Jul 28 17:26:12.526733 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.526734 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.527046 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527027 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} err="failed to get container status \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" Jul 28 17:26:12.527096 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527047 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.527313 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527293 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} err="failed to get container status \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" Jul 28 17:26:12.527376 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527313 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.527376 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527476 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/425ffd7c-0cd3-447d-91b6-1605ace2af7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527476 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/425ffd7c-0cd3-447d-91b6-1605ace2af7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527552 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527552 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527552 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527529 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} err="failed to get container status \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" Jul 28 17:26:12.527552 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527547 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.527718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527718 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527711 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527775 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} err="failed to get container status \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527796 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.527939 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.528351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.528351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.527980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5wfx\" (UniqueName: \"kubernetes.io/projected/425ffd7c-0cd3-447d-91b6-1605ace2af7a-kube-api-access-f5wfx\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.528351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-config\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.528351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.528351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528109 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} err="failed to get container status \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" Jul 28 17:26:12.528351 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528125 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.528688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528430 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} err="failed to get container status \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" Jul 28 17:26:12.528688 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528451 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.528829 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528805 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} err="failed to get container status \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" Jul 28 17:26:12.528891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.528832 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.529650 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.529625 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} err="failed to get container status \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" Jul 28 17:26:12.529650 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.529651 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.530168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530141 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} err="failed to get container status \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" Jul 28 17:26:12.530168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530164 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.530436 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530416 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} err="failed to get container status \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" Jul 28 17:26:12.530510 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530438 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.530670 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530651 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} err="failed to get container status \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" Jul 28 17:26:12.530713 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530671 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.530917 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530899 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} err="failed to get container status \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" Jul 28 17:26:12.530988 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.530918 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.531164 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531144 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} err="failed to get container status \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" Jul 28 17:26:12.531164 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531162 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.531422 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531401 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} err="failed to get container status \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" Jul 28 17:26:12.531492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531424 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.531680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531654 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} err="failed to get container status \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" Jul 28 17:26:12.531680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531680 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.531965 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531947 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} err="failed to get container status \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" Jul 28 17:26:12.532026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531967 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.532026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.531968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:26:12.532247 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.532226 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} err="failed to get container status \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" Jul 28 17:26:12.532301 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.532249 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.532495 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.532478 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} err="failed to get container status \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" Jul 28 17:26:12.532555 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.532495 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.532777 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.532742 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} err="failed to get container status \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" Jul 28 17:26:12.532855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.532780 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.533065 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533036 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} err="failed to get container status \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" Jul 28 17:26:12.533065 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533065 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.533345 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533322 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} err="failed to get container status \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" Jul 28 17:26:12.533404 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533356 2564 scope.go:117] "RemoveContainer" containerID="e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051" Jul 28 17:26:12.533608 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533587 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051"} err="failed to get container status \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": rpc error: code = NotFound desc = could not find container \"e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051\": container with ID starting with e877d53b9c0c482ea271b408239041e23562c271396620f19b83fe2676f1e051 not found: ID does not exist" Jul 28 17:26:12.533703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533607 2564 scope.go:117] "RemoveContainer" containerID="3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5" Jul 28 17:26:12.533922 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533889 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5"} err="failed to get container status \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": rpc error: code = NotFound desc = could not find container \"3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5\": container with ID starting with 3cbb667792880aea6e44de4f4aef3dd8de6c7083f441c15a7ed89d5595cdfac5 not found: ID does not exist" Jul 28 17:26:12.534006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.533923 2564 scope.go:117] "RemoveContainer" containerID="85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e" Jul 28 17:26:12.534202 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.534184 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e"} err="failed to get container status \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": rpc error: code = NotFound desc = could not find container \"85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e\": container with ID starting with 85893b7aebcfae3c2b8d9391bb32ce604c638105b94f082d64e2ab042ff7318e not found: ID does not exist" Jul 28 17:26:12.534202 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.534201 2564 scope.go:117] "RemoveContainer" containerID="f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1" Jul 28 17:26:12.534487 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.534467 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1"} err="failed to get container status \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": rpc error: code = NotFound desc = could not find container \"f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1\": container with ID starting with f30dfe860726895883f4d15bcdc235da5f1c18981d153a8cace48109f03a89c1 not found: ID does not exist" Jul 28 17:26:12.534487 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.534485 2564 scope.go:117] "RemoveContainer" containerID="735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e" Jul 28 17:26:12.534790 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.534769 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e"} err="failed to get container status \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": rpc error: code = NotFound desc = could not find container \"735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e\": container with ID starting with 735ea292fa65db37523f0e4ead50ea900a78534f30aa48ecea5b880277a4a69e not found: ID does not exist" Jul 28 17:26:12.534790 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.534789 2564 scope.go:117] "RemoveContainer" containerID="ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191" Jul 28 17:26:12.535036 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.535017 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191"} err="failed to get container status \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": rpc error: code = NotFound desc = could not find container \"ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191\": container with ID starting with ad420e6c3ef831f92b1cdd588be68b4d881ef015a2a52ae054ebaa788b5d6191 not found: ID does not exist" Jul 28 17:26:12.535090 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.535037 2564 scope.go:117] "RemoveContainer" containerID="4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878" Jul 28 17:26:12.535263 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.535243 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878"} err="failed to get container status \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": rpc error: code = NotFound desc = could not find container \"4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878\": container with ID starting with 4ed99ee7f80e1faf463bdee714715893f34279db6fd5958d6a99647d38943878 not found: ID does not exist" Jul 28 17:26:12.628535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.628879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.628708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.629402 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5wfx\" (UniqueName: \"kubernetes.io/projected/425ffd7c-0cd3-447d-91b6-1605ace2af7a-kube-api-access-f5wfx\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-config\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/425ffd7c-0cd3-447d-91b6-1605ace2af7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/425ffd7c-0cd3-447d-91b6-1605ace2af7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.629887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.630139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.630494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.630178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.631441 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.630751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.631772 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.631733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.632819 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.632201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.632819 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.632729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.633301 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.633276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.633386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.633346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/425ffd7c-0cd3-447d-91b6-1605ace2af7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634326 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634402 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/425ffd7c-0cd3-447d-91b6-1605ace2af7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634502 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634481 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/425ffd7c-0cd3-447d-91b6-1605ace2af7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634548 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.634792 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.634747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/425ffd7c-0cd3-447d-91b6-1605ace2af7a-config\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.643807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.641162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5wfx\" (UniqueName: \"kubernetes.io/projected/425ffd7c-0cd3-447d-91b6-1605ace2af7a-kube-api-access-f5wfx\") pod \"prometheus-k8s-0\" (UID: \"425ffd7c-0cd3-447d-91b6-1605ace2af7a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.828026 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.827983 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:26:12.974230 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:12.974204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:26:12.975702 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:26:12.975679 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425ffd7c_0cd3_447d_91b6_1605ace2af7a.slice/crio-35a1a3765d280d4fbbca56aa541f6c53780518667b001b57ef78b4c9b0474c5a WatchSource:0}: Error finding container 35a1a3765d280d4fbbca56aa541f6c53780518667b001b57ef78b4c9b0474c5a: Status 404 returned error can't find the container with id 35a1a3765d280d4fbbca56aa541f6c53780518667b001b57ef78b4c9b0474c5a Jul 28 17:26:13.376035 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:13.376001 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca62eb39-29c0-4ca3-99ed-f09e2a6013a1" path="/var/lib/kubelet/pods/ca62eb39-29c0-4ca3-99ed-f09e2a6013a1/volumes" Jul 28 17:26:13.448506 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:13.448471 2564 generic.go:356] "Generic (PLEG): container finished" podID="425ffd7c-0cd3-447d-91b6-1605ace2af7a" containerID="f47f10ed28944937752be51704f262b2690a2e0857e381443eb0dd7a6a8fa1ad" exitCode=0 Jul 28 17:26:13.448922 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:13.448554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerDied","Data":"f47f10ed28944937752be51704f262b2690a2e0857e381443eb0dd7a6a8fa1ad"} Jul 28 17:26:13.448922 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:13.448619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"35a1a3765d280d4fbbca56aa541f6c53780518667b001b57ef78b4c9b0474c5a"} Jul 28 17:26:14.459730 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:14.459694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"25595f9bc56230c59a2591a6465201dffc6c6fdbf6a4c9022653ec4df04cfa12"} Jul 28 17:26:14.460118 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:14.459771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"540e7ebf168e8742a8df8b8f694fb7d084f86f5f17c4481459d20e9d1e887527"} Jul 28 17:26:14.460118 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:14.459801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"769cfe0b367dfd4fb038ed675e481eb0e7be8bd3cb7a058b90ec9df24574eaf1"} Jul 28 17:26:14.460118 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:14.459825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"5a641df76d2197cecebf096a7eca3f73263c5c7998b701cc2da0756bfd63aa78"} Jul 28 17:26:14.460118 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:14.459848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"8e0d171c6a1c5f51a967d9f081148e7915b0ac5a54f036da3baf1b23e293aa45"} Jul 28 17:26:14.460118 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:14.459871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"425ffd7c-0cd3-447d-91b6-1605ace2af7a","Type":"ContainerStarted","Data":"4f1a08754f57d0e769423069f566ee6ca5e40aa00bf92f3dbd1a6aa581847ae4"} Jul 28 17:26:17.828585 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:26:17.828545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:27:12.828624 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:12.828528 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:27:12.844572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:12.844546 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:27:12.884845 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:12.884786 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=60.864108958 podStartE2EDuration="1m0.884752599s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.643641ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:14.509302951 +0000 UTC m=+207.661803127" watchObservedRunningTime="2026-07-28 17:27:12.884752599 +0000 UTC m=+266.037252794" Jul 28 17:27:13.726905 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:13.726876 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:27:47.308451 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:47.308422 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:27:47.327805 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:47.327772 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:27:47.332709 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:27:47.332679 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 17:30:35.007251 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.007205 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk"] Jul 28 17:30:35.010679 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.010650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.015248 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.015223 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tpxv6\"" Jul 28 17:30:35.015677 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.015662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 17:30:35.015994 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.015980 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 17:30:35.028426 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.028391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5758q\" (UniqueName: \"kubernetes.io/projected/08f85ac5-3a5e-463a-8dda-b59c1c543594-kube-api-access-5758q\") pod \"cert-manager-cainjector-559cdd8c87-cdbmk\" (UID: \"08f85ac5-3a5e-463a-8dda-b59c1c543594\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.028572 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.028501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08f85ac5-3a5e-463a-8dda-b59c1c543594-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cdbmk\" (UID: \"08f85ac5-3a5e-463a-8dda-b59c1c543594\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.028691 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.028676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk"] Jul 28 17:30:35.129661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.129627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08f85ac5-3a5e-463a-8dda-b59c1c543594-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cdbmk\" (UID: \"08f85ac5-3a5e-463a-8dda-b59c1c543594\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.129867 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.129706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5758q\" (UniqueName: \"kubernetes.io/projected/08f85ac5-3a5e-463a-8dda-b59c1c543594-kube-api-access-5758q\") pod \"cert-manager-cainjector-559cdd8c87-cdbmk\" (UID: \"08f85ac5-3a5e-463a-8dda-b59c1c543594\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.138980 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.138955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08f85ac5-3a5e-463a-8dda-b59c1c543594-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cdbmk\" (UID: \"08f85ac5-3a5e-463a-8dda-b59c1c543594\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.139117 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.139087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5758q\" (UniqueName: \"kubernetes.io/projected/08f85ac5-3a5e-463a-8dda-b59c1c543594-kube-api-access-5758q\") pod \"cert-manager-cainjector-559cdd8c87-cdbmk\" (UID: \"08f85ac5-3a5e-463a-8dda-b59c1c543594\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.353395 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.353362 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" Jul 28 17:30:35.486314 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.486251 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk"] Jul 28 17:30:35.488606 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:30:35.488579 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08f85ac5_3a5e_463a_8dda_b59c1c543594.slice/crio-0a4149e44c20609b8ed05e5b49fbdfb4e90c791ad03ca16770bd5848b596153b WatchSource:0}: Error finding container 0a4149e44c20609b8ed05e5b49fbdfb4e90c791ad03ca16770bd5848b596153b: Status 404 returned error can't find the container with id 0a4149e44c20609b8ed05e5b49fbdfb4e90c791ad03ca16770bd5848b596153b Jul 28 17:30:35.490585 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.490568 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:30:35.517603 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:35.517560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" event={"ID":"08f85ac5-3a5e-463a-8dda-b59c1c543594","Type":"ContainerStarted","Data":"0a4149e44c20609b8ed05e5b49fbdfb4e90c791ad03ca16770bd5848b596153b"} Jul 28 17:30:39.537309 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:39.537274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" event={"ID":"08f85ac5-3a5e-463a-8dda-b59c1c543594","Type":"ContainerStarted","Data":"5d74ce3b8f195a045c87cffcd5b499926c8d4c5b8d9912e775ef128e342c9a1d"} Jul 28 17:30:39.557637 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:39.557586 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cdbmk" podStartSLOduration=2.5152284700000003 podStartE2EDuration="5.557567735s" totalImagesPullingTime="3.042339265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:39.554967107 +0000 UTC m=+472.707467282" watchObservedRunningTime="2026-07-28 17:30:39.557567735 +0000 UTC m=+472.710067911" Jul 28 17:30:41.517324 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.517286 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pnwj4"] Jul 28 17:30:41.520877 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.520854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.523851 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.523829 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2sn7t\"" Jul 28 17:30:41.531243 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.531218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pnwj4"] Jul 28 17:30:41.590156 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.590110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9k5\" (UniqueName: \"kubernetes.io/projected/7faab0df-f17a-47bb-85d0-1111fe43c362-kube-api-access-9v9k5\") pod \"cert-manager-55b4d954d5-pnwj4\" (UID: \"7faab0df-f17a-47bb-85d0-1111fe43c362\") " pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.590349 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.590190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7faab0df-f17a-47bb-85d0-1111fe43c362-bound-sa-token\") pod \"cert-manager-55b4d954d5-pnwj4\" (UID: \"7faab0df-f17a-47bb-85d0-1111fe43c362\") " pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.690744 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.690709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7faab0df-f17a-47bb-85d0-1111fe43c362-bound-sa-token\") pod \"cert-manager-55b4d954d5-pnwj4\" (UID: \"7faab0df-f17a-47bb-85d0-1111fe43c362\") " pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.690919 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.690805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9k5\" (UniqueName: \"kubernetes.io/projected/7faab0df-f17a-47bb-85d0-1111fe43c362-kube-api-access-9v9k5\") pod \"cert-manager-55b4d954d5-pnwj4\" (UID: \"7faab0df-f17a-47bb-85d0-1111fe43c362\") " pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.700340 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.700310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7faab0df-f17a-47bb-85d0-1111fe43c362-bound-sa-token\") pod \"cert-manager-55b4d954d5-pnwj4\" (UID: \"7faab0df-f17a-47bb-85d0-1111fe43c362\") " pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.700452 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.700370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9k5\" (UniqueName: \"kubernetes.io/projected/7faab0df-f17a-47bb-85d0-1111fe43c362-kube-api-access-9v9k5\") pod \"cert-manager-55b4d954d5-pnwj4\" (UID: \"7faab0df-f17a-47bb-85d0-1111fe43c362\") " pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.830810 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.830753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pnwj4" Jul 28 17:30:41.960956 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:41.960934 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pnwj4"] Jul 28 17:30:41.963366 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:30:41.963327 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7faab0df_f17a_47bb_85d0_1111fe43c362.slice/crio-c0cab0f92bf19e8afffce33e92a788a639a184588f14ee0d44217c1db29f7f49 WatchSource:0}: Error finding container c0cab0f92bf19e8afffce33e92a788a639a184588f14ee0d44217c1db29f7f49: Status 404 returned error can't find the container with id c0cab0f92bf19e8afffce33e92a788a639a184588f14ee0d44217c1db29f7f49 Jul 28 17:30:42.551784 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:42.551731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pnwj4" event={"ID":"7faab0df-f17a-47bb-85d0-1111fe43c362","Type":"ContainerStarted","Data":"b4e26cdf390205da2c2aad13c00c6e6e81eebf849e6300f68026eece13d4323e"} Jul 28 17:30:42.551784 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:42.551790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pnwj4" event={"ID":"7faab0df-f17a-47bb-85d0-1111fe43c362","Type":"ContainerStarted","Data":"c0cab0f92bf19e8afffce33e92a788a639a184588f14ee0d44217c1db29f7f49"} Jul 28 17:30:42.570478 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:30:42.570428 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pnwj4" podStartSLOduration=1.5704099569999999 podStartE2EDuration="1.570409957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:42.569248475 +0000 UTC m=+475.721748652" watchObservedRunningTime="2026-07-28 17:30:42.570409957 +0000 UTC m=+475.722910132" Jul 28 17:31:24.293949 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.293909 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7"] Jul 28 17:31:24.299368 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.299345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.304724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.304684 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7lw22\"" Jul 28 17:31:24.304724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.304709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 17:31:24.304724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.304716 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 17:31:24.304724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.304725 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:31:24.305047 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.304719 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 17:31:24.305047 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.304798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 17:31:24.313887 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.313858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7"] Jul 28 17:31:24.362318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.362278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-manager-config\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.362519 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.362446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-metrics-cert\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.362519 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.362481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplc8\" (UniqueName: \"kubernetes.io/projected/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-kube-api-access-hplc8\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.362519 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.362509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-cert\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.463895 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.463844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplc8\" (UniqueName: \"kubernetes.io/projected/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-kube-api-access-hplc8\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.463895 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.463893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-cert\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.464105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.463937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-manager-config\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.464105 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.464070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-metrics-cert\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.464721 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.464694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-manager-config\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.466541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.466511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-cert\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.466636 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.466577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-metrics-cert\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.473581 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.473552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplc8\" (UniqueName: \"kubernetes.io/projected/b84f41a6-b6ee-41ce-86bc-27fabb9352c6-kube-api-access-hplc8\") pod \"lws-controller-manager-fd6565d47-n59f7\" (UID: \"b84f41a6-b6ee-41ce-86bc-27fabb9352c6\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.610084 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.610045 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:24.787949 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:24.787918 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7"] Jul 28 17:31:24.789163 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:31:24.789135 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84f41a6_b6ee_41ce_86bc_27fabb9352c6.slice/crio-c1c8fd5e93f441404936d1144ba21b2faf58bc29589339d75837e71daa9e6f47 WatchSource:0}: Error finding container c1c8fd5e93f441404936d1144ba21b2faf58bc29589339d75837e71daa9e6f47: Status 404 returned error can't find the container with id c1c8fd5e93f441404936d1144ba21b2faf58bc29589339d75837e71daa9e6f47 Jul 28 17:31:25.740238 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:25.740162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" event={"ID":"b84f41a6-b6ee-41ce-86bc-27fabb9352c6","Type":"ContainerStarted","Data":"c1c8fd5e93f441404936d1144ba21b2faf58bc29589339d75837e71daa9e6f47"} Jul 28 17:31:27.751092 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:27.751047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" event={"ID":"b84f41a6-b6ee-41ce-86bc-27fabb9352c6","Type":"ContainerStarted","Data":"7168727fa5363674be5110a5f5c98312d872795035ea8369c3d281e6880724de"} Jul 28 17:31:27.751461 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:27.751143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:27.780645 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:27.780591 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" podStartSLOduration=1.480133095 podStartE2EDuration="3.780575073s" totalImagesPullingTime="2.300441978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:27.778198042 +0000 UTC m=+520.930698247" watchObservedRunningTime="2026-07-28 17:31:27.780575073 +0000 UTC m=+520.933075249" Jul 28 17:31:38.757911 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:38.757871 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-n59f7" Jul 28 17:31:55.811504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.811409 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt"] Jul 28 17:31:55.815048 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.815023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:55.818287 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.818263 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gdhth\"" Jul 28 17:31:55.818446 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.818290 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 17:31:55.818446 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.818262 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 17:31:55.827214 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.827183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt"] Jul 28 17:31:55.953694 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.953653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xh8l\" (UniqueName: \"kubernetes.io/projected/42608b8a-a47a-4ce6-a66b-48f09e393ff9-kube-api-access-6xh8l\") pod \"kuadrant-operator-controller-manager-66866f5fcc-4zxvt\" (UID: \"42608b8a-a47a-4ce6-a66b-48f09e393ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:55.953903 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:55.953709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42608b8a-a47a-4ce6-a66b-48f09e393ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-4zxvt\" (UID: \"42608b8a-a47a-4ce6-a66b-48f09e393ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:56.054816 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.054777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xh8l\" (UniqueName: \"kubernetes.io/projected/42608b8a-a47a-4ce6-a66b-48f09e393ff9-kube-api-access-6xh8l\") pod \"kuadrant-operator-controller-manager-66866f5fcc-4zxvt\" (UID: \"42608b8a-a47a-4ce6-a66b-48f09e393ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:56.054962 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.054830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42608b8a-a47a-4ce6-a66b-48f09e393ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-4zxvt\" (UID: \"42608b8a-a47a-4ce6-a66b-48f09e393ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:56.055229 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.055208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42608b8a-a47a-4ce6-a66b-48f09e393ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-4zxvt\" (UID: \"42608b8a-a47a-4ce6-a66b-48f09e393ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:56.068523 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.068459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xh8l\" (UniqueName: \"kubernetes.io/projected/42608b8a-a47a-4ce6-a66b-48f09e393ff9-kube-api-access-6xh8l\") pod \"kuadrant-operator-controller-manager-66866f5fcc-4zxvt\" (UID: \"42608b8a-a47a-4ce6-a66b-48f09e393ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:56.126833 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.126786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:31:56.275299 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.275272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt"] Jul 28 17:31:56.277003 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:31:56.276970 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42608b8a_a47a_4ce6_a66b_48f09e393ff9.slice/crio-da54ea0facdf56389a656f6ff236430cd11e523b511c6c695c5426d1c14667db WatchSource:0}: Error finding container da54ea0facdf56389a656f6ff236430cd11e523b511c6c695c5426d1c14667db: Status 404 returned error can't find the container with id da54ea0facdf56389a656f6ff236430cd11e523b511c6c695c5426d1c14667db Jul 28 17:31:56.887852 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:31:56.887813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" event={"ID":"42608b8a-a47a-4ce6-a66b-48f09e393ff9","Type":"ContainerStarted","Data":"da54ea0facdf56389a656f6ff236430cd11e523b511c6c695c5426d1c14667db"} Jul 28 17:32:00.913370 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:00.913324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" event={"ID":"42608b8a-a47a-4ce6-a66b-48f09e393ff9","Type":"ContainerStarted","Data":"7b7e2ab62f049c276743c55cd2372116daa127f897c25ba83c0ff3eda3501c48"} Jul 28 17:32:00.913793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:00.913412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:32:00.948625 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:00.948552 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" podStartSLOduration=2.119205894 podStartE2EDuration="5.948529337s" totalImagesPullingTime="3.829323443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:00.946691893 +0000 UTC m=+554.099192070" watchObservedRunningTime="2026-07-28 17:32:00.948529337 +0000 UTC m=+554.101029515" Jul 28 17:32:02.884729 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:02.884687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj"] Jul 28 17:32:02.892951 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:02.892919 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:02.895988 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:02.895961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-74x2r\"" Jul 28 17:32:02.896186 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:02.895961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 28 17:32:02.897260 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:02.897205 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 28 17:32:02.899679 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:02.899641 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj"] Jul 28 17:32:03.025237 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.025195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbtz\" (UniqueName: \"kubernetes.io/projected/492fed35-d0ad-40b6-af96-fedd3e3c4b64-kube-api-access-6jbtz\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.025237 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.025246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/492fed35-d0ad-40b6-af96-fedd3e3c4b64-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.025457 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.025319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/492fed35-d0ad-40b6-af96-fedd3e3c4b64-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.032995 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.032956 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm"] Jul 28 17:32:03.036775 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.036722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:03.039677 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.039653 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fbs4j\"" Jul 28 17:32:03.039813 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.039653 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 28 17:32:03.047269 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.047242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm"] Jul 28 17:32:03.126723 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.126679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbtz\" (UniqueName: \"kubernetes.io/projected/492fed35-d0ad-40b6-af96-fedd3e3c4b64-kube-api-access-6jbtz\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.126928 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.126736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/492fed35-d0ad-40b6-af96-fedd3e3c4b64-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.126928 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.126815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/492fed35-d0ad-40b6-af96-fedd3e3c4b64-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.127007 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:32:03.126971 2564 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 28 17:32:03.127063 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:32:03.127052 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492fed35-d0ad-40b6-af96-fedd3e3c4b64-plugin-serving-cert podName:492fed35-d0ad-40b6-af96-fedd3e3c4b64 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:03.627030244 +0000 UTC m=+556.779530398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/492fed35-d0ad-40b6-af96-fedd3e3c4b64-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-99wjj" (UID: "492fed35-d0ad-40b6-af96-fedd3e3c4b64") : secret "plugin-serving-cert" not found Jul 28 17:32:03.127459 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.127440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/492fed35-d0ad-40b6-af96-fedd3e3c4b64-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.136976 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.136904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbtz\" (UniqueName: \"kubernetes.io/projected/492fed35-d0ad-40b6-af96-fedd3e3c4b64-kube-api-access-6jbtz\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.227522 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.227489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6hr\" (UniqueName: \"kubernetes.io/projected/d1b822ff-8d7b-4510-aa67-0ff38770843b-kube-api-access-pj6hr\") pod \"dns-operator-controller-manager-ccbdf686-nkbjm\" (UID: \"d1b822ff-8d7b-4510-aa67-0ff38770843b\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:03.328787 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.328721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6hr\" (UniqueName: \"kubernetes.io/projected/d1b822ff-8d7b-4510-aa67-0ff38770843b-kube-api-access-pj6hr\") pod \"dns-operator-controller-manager-ccbdf686-nkbjm\" (UID: \"d1b822ff-8d7b-4510-aa67-0ff38770843b\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:03.340529 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.340501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6hr\" (UniqueName: \"kubernetes.io/projected/d1b822ff-8d7b-4510-aa67-0ff38770843b-kube-api-access-pj6hr\") pod \"dns-operator-controller-manager-ccbdf686-nkbjm\" (UID: \"d1b822ff-8d7b-4510-aa67-0ff38770843b\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:03.347248 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.347210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:03.487591 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.487565 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm"] Jul 28 17:32:03.489638 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:32:03.489614 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b822ff_8d7b_4510_aa67_0ff38770843b.slice/crio-ca0d8ab2539d5d8eef9fbcebd0ec3cef6b09c1ca0a13c2c16d21dc64bea470aa WatchSource:0}: Error finding container ca0d8ab2539d5d8eef9fbcebd0ec3cef6b09c1ca0a13c2c16d21dc64bea470aa: Status 404 returned error can't find the container with id ca0d8ab2539d5d8eef9fbcebd0ec3cef6b09c1ca0a13c2c16d21dc64bea470aa Jul 28 17:32:03.631210 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.631172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/492fed35-d0ad-40b6-af96-fedd3e3c4b64-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.633675 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.633642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/492fed35-d0ad-40b6-af96-fedd3e3c4b64-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-99wjj\" (UID: \"492fed35-d0ad-40b6-af96-fedd3e3c4b64\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.809835 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.809710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" Jul 28 17:32:03.929017 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.928979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" event={"ID":"d1b822ff-8d7b-4510-aa67-0ff38770843b","Type":"ContainerStarted","Data":"ca0d8ab2539d5d8eef9fbcebd0ec3cef6b09c1ca0a13c2c16d21dc64bea470aa"} Jul 28 17:32:03.952630 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:03.952600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj"] Jul 28 17:32:03.954360 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:32:03.954327 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod492fed35_d0ad_40b6_af96_fedd3e3c4b64.slice/crio-b1760e4bac9bb0edaf4cf6785086902f2e72c0f65dd9268069122dbd7f1c45ba WatchSource:0}: Error finding container b1760e4bac9bb0edaf4cf6785086902f2e72c0f65dd9268069122dbd7f1c45ba: Status 404 returned error can't find the container with id b1760e4bac9bb0edaf4cf6785086902f2e72c0f65dd9268069122dbd7f1c45ba Jul 28 17:32:04.936690 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:04.936646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" event={"ID":"492fed35-d0ad-40b6-af96-fedd3e3c4b64","Type":"ContainerStarted","Data":"b1760e4bac9bb0edaf4cf6785086902f2e72c0f65dd9268069122dbd7f1c45ba"} Jul 28 17:32:05.944548 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:05.944510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" event={"ID":"d1b822ff-8d7b-4510-aa67-0ff38770843b","Type":"ContainerStarted","Data":"48102aeecb0d0a964486a06f9c200f99ec7cfa3aa137973644292205ce0a3222"} Jul 28 17:32:05.945073 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:05.944633 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:05.964663 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:05.964590 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" podStartSLOduration=1.427451756 podStartE2EDuration="2.964569374s" totalImagesPullingTime="1.537117618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:05.963426067 +0000 UTC m=+559.115926263" watchObservedRunningTime="2026-07-28 17:32:05.964569374 +0000 UTC m=+559.117069554" Jul 28 17:32:09.969568 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:09.969528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" event={"ID":"492fed35-d0ad-40b6-af96-fedd3e3c4b64","Type":"ContainerStarted","Data":"78e38993d6b46401c5e76b008e13e3d779302148caa1b5e6fec0520958f0ea70"} Jul 28 17:32:09.990745 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:09.990684 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-99wjj" podStartSLOduration=3.054885358 podStartE2EDuration="7.990666241s" totalImagesPullingTime="4.935780883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:09.99014827 +0000 UTC m=+563.142648448" watchObservedRunningTime="2026-07-28 17:32:09.990666241 +0000 UTC m=+563.143166433" Jul 28 17:32:11.919900 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:11.919867 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-4zxvt" Jul 28 17:32:16.952113 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:16.952084 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-nkbjm" Jul 28 17:32:46.846670 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:46.846629 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zllks"] Jul 28 17:32:46.853017 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:46.852990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:46.855655 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:46.855631 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 17:32:46.859219 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:46.859193 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zllks"] Jul 28 17:32:46.894838 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:46.894801 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zllks"] Jul 28 17:32:47.028185 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.028133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpftp\" (UniqueName: \"kubernetes.io/projected/6893502a-4823-4003-91d2-70c3fa5dedeb-kube-api-access-zpftp\") pod \"limitador-limitador-85d9db4d5d-zllks\" (UID: \"6893502a-4823-4003-91d2-70c3fa5dedeb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.028369 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.028199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6893502a-4823-4003-91d2-70c3fa5dedeb-config-file\") pod \"limitador-limitador-85d9db4d5d-zllks\" (UID: \"6893502a-4823-4003-91d2-70c3fa5dedeb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.129601 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.129500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6893502a-4823-4003-91d2-70c3fa5dedeb-config-file\") pod \"limitador-limitador-85d9db4d5d-zllks\" (UID: \"6893502a-4823-4003-91d2-70c3fa5dedeb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.129795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.129646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpftp\" (UniqueName: \"kubernetes.io/projected/6893502a-4823-4003-91d2-70c3fa5dedeb-kube-api-access-zpftp\") pod \"limitador-limitador-85d9db4d5d-zllks\" (UID: \"6893502a-4823-4003-91d2-70c3fa5dedeb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.130280 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.130260 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6893502a-4823-4003-91d2-70c3fa5dedeb-config-file\") pod \"limitador-limitador-85d9db4d5d-zllks\" (UID: \"6893502a-4823-4003-91d2-70c3fa5dedeb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.138983 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.138954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpftp\" (UniqueName: \"kubernetes.io/projected/6893502a-4823-4003-91d2-70c3fa5dedeb-kube-api-access-zpftp\") pod \"limitador-limitador-85d9db4d5d-zllks\" (UID: \"6893502a-4823-4003-91d2-70c3fa5dedeb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.164854 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.164807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:47.321112 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.319490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zllks"] Jul 28 17:32:47.323752 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:32:47.323714 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6893502a_4823_4003_91d2_70c3fa5dedeb.slice/crio-3e9bee29f9cf4e6d5a47331d6cd9bb03059eb52eaa50cde4d71e82a6d58c9bb5 WatchSource:0}: Error finding container 3e9bee29f9cf4e6d5a47331d6cd9bb03059eb52eaa50cde4d71e82a6d58c9bb5: Status 404 returned error can't find the container with id 3e9bee29f9cf4e6d5a47331d6cd9bb03059eb52eaa50cde4d71e82a6d58c9bb5 Jul 28 17:32:47.355468 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.355434 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:32:47.359213 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:47.359186 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:32:48.158116 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:48.158083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" event={"ID":"6893502a-4823-4003-91d2-70c3fa5dedeb","Type":"ContainerStarted","Data":"3e9bee29f9cf4e6d5a47331d6cd9bb03059eb52eaa50cde4d71e82a6d58c9bb5"} Jul 28 17:32:49.164077 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:49.164040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" event={"ID":"6893502a-4823-4003-91d2-70c3fa5dedeb","Type":"ContainerStarted","Data":"b4d02627ccef3a6931d79097ee73fc0545b84e951f57cfa9efbd0d14e9076990"} Jul 28 17:32:49.164518 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:49.164167 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:32:49.183816 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:32:49.183742 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" podStartSLOduration=1.857513309 podStartE2EDuration="3.183725251s" totalImagesPullingTime="1.326211942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:49.183349147 +0000 UTC m=+602.335849323" watchObservedRunningTime="2026-07-28 17:32:49.183725251 +0000 UTC m=+602.336225428" Jul 28 17:33:00.169813 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:33:00.169746 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zllks" Jul 28 17:35:30.315652 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.315616 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f56bf77c-cn6qr"] Jul 28 17:35:30.319696 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.319670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.322768 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.322723 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 17:35:30.324099 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.324079 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 17:35:30.324099 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.324095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 17:35:30.324275 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.324243 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 17:35:30.324372 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.324356 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 17:35:30.324438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.324386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-frkgx\"" Jul 28 17:35:30.329868 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.329834 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 17:35:30.332238 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.332212 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f56bf77c-cn6qr"] Jul 28 17:35:30.463751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.463709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-trusted-ca-bundle\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.463751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.463748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62z69\" (UniqueName: \"kubernetes.io/projected/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-kube-api-access-62z69\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.463991 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.463844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-serving-cert\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.463991 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.463871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-oauth-serving-cert\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.463991 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.463894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-config\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.464138 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.463985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-service-ca\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.464138 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.464025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-oauth-config\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565563 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-trusted-ca-bundle\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565563 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62z69\" (UniqueName: \"kubernetes.io/projected/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-kube-api-access-62z69\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565835 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-serving-cert\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565835 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-oauth-serving-cert\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565919 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-config\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565919 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-service-ca\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.565983 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.565929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-oauth-config\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.566505 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.566478 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-oauth-serving-cert\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.566684 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.566524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-service-ca\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.566684 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.566595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-trusted-ca-bundle\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.566840 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.566683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-config\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.568380 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.568357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-oauth-config\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.568479 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.568455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-console-serving-cert\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.576260 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.576236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62z69\" (UniqueName: \"kubernetes.io/projected/7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646-kube-api-access-62z69\") pod \"console-76f56bf77c-cn6qr\" (UID: \"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646\") " pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.630218 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.630177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:30.778405 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.778378 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f56bf77c-cn6qr"] Jul 28 17:35:30.782791 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:35:30.782741 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee7af80_c0c4_4bc4_b2ef_b9ca29c09646.slice/crio-137ccc3e3cf5d9725e38f381fef7454fbeb06dc51ba37302210527dd2026a595 WatchSource:0}: Error finding container 137ccc3e3cf5d9725e38f381fef7454fbeb06dc51ba37302210527dd2026a595: Status 404 returned error can't find the container with id 137ccc3e3cf5d9725e38f381fef7454fbeb06dc51ba37302210527dd2026a595 Jul 28 17:35:30.949576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.949542 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f56bf77c-cn6qr" event={"ID":"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646","Type":"ContainerStarted","Data":"514ff2187740f17535176b33c5a655359c45b7ec71f9a4c84286d7768fa548e3"} Jul 28 17:35:30.949744 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.949591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f56bf77c-cn6qr" event={"ID":"7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646","Type":"ContainerStarted","Data":"137ccc3e3cf5d9725e38f381fef7454fbeb06dc51ba37302210527dd2026a595"} Jul 28 17:35:30.970802 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:30.970733 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76f56bf77c-cn6qr" podStartSLOduration=0.970717308 podStartE2EDuration="970.717308ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:30.970063885 +0000 UTC m=+764.122564079" watchObservedRunningTime="2026-07-28 17:35:30.970717308 +0000 UTC m=+764.123217484" Jul 28 17:35:40.630587 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:40.630490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:40.630587 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:40.630577 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:40.635444 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:40.635416 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:35:41.003956 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:35:41.003880 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f56bf77c-cn6qr" Jul 28 17:36:13.109046 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.108955 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj"] Jul 28 17:36:13.112550 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.112525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.115959 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.115930 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\"" Jul 28 17:36:13.116114 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.116031 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\"" Jul 28 17:36:13.117222 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.117191 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\"" Jul 28 17:36:13.117360 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.117318 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-dcvlk\"" Jul 28 17:36:13.125516 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.125488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj"] Jul 28 17:36:13.245201 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.245161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-dshm\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.245201 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.245210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.245438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.245288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whwq\" (UniqueName: \"kubernetes.io/projected/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-kube-api-access-4whwq\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.245438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.245328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.245438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.245345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.245438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.245399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-home\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.346648 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.346606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-dshm\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.346648 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.346651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.346937 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.346684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whwq\" (UniqueName: \"kubernetes.io/projected/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-kube-api-access-4whwq\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.346937 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.346707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.346937 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.346725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.346937 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.346744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-home\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.347145 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.347129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.347207 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.347181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-home\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.347207 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.347196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.349132 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.349107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-dshm\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.349275 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.349254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.356151 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.356123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whwq\" (UniqueName: \"kubernetes.io/projected/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-kube-api-access-4whwq\") pod \"fc-smoke-test-kserve-7f4687897-4mtcj\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.423528 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.423433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:13.574348 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.574320 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj"] Jul 28 17:36:13.576586 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:36:13.576552 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a2fe5d_fba4_4b35_ac30_c9133afbdede.slice/crio-d02d5b69789fed033eac4a8350d31726d097eb943217c382799aaa2571f44b48 WatchSource:0}: Error finding container d02d5b69789fed033eac4a8350d31726d097eb943217c382799aaa2571f44b48: Status 404 returned error can't find the container with id d02d5b69789fed033eac4a8350d31726d097eb943217c382799aaa2571f44b48 Jul 28 17:36:13.578456 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:13.578437 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:36:14.160188 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:14.160153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" event={"ID":"c9a2fe5d-fba4-4b35-ac30-c9133afbdede","Type":"ContainerStarted","Data":"d02d5b69789fed033eac4a8350d31726d097eb943217c382799aaa2571f44b48"} Jul 28 17:36:16.179820 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:16.179779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" event={"ID":"c9a2fe5d-fba4-4b35-ac30-c9133afbdede","Type":"ContainerStarted","Data":"65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3"} Jul 28 17:36:16.198070 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:16.197999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" podStartSLOduration=1.5091463809999999 podStartE2EDuration="3.197977093s" totalImagesPullingTime="1.688830712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:16.197721068 +0000 UTC m=+809.350221245" watchObservedRunningTime="2026-07-28 17:36:16.197977093 +0000 UTC m=+809.350477271" Jul 28 17:36:16.424050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:16.424008 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:23.424104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:23.424039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:23.436778 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:23.436735 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:24.232308 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:24.232270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:54.441670 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.441632 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj"] Jul 28 17:36:54.451444 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:36:54.451408 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:36:54.451444 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:36:54.451444 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:36:54.451444 ip-10-0-131-29 kubenswrapper[2564]: > containerID="65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" cmd=["/bin/sleep","15"] Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.451464 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" message="" err=< Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:36:54.451480 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" podUID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" containerName="main" containerID="cri-o://65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" Jul 28 17:36:54.451713 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.451517 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" podUID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" containerName="main" containerID="cri-o://65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" gracePeriod=60 Jul 28 17:36:54.811029 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.811004 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:54.940292 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940258 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-home\") pod \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " Jul 28 17:36:54.940492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-dshm\") pod \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " Jul 28 17:36:54.940492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940381 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-model-cache\") pod \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " Jul 28 17:36:54.940492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940411 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tmp-dir\") pod \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " Jul 28 17:36:54.940492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940433 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tls-certs\") pod \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " Jul 28 17:36:54.940492 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940470 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4whwq\" (UniqueName: \"kubernetes.io/projected/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-kube-api-access-4whwq\") pod \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\" (UID: \"c9a2fe5d-fba4-4b35-ac30-c9133afbdede\") " Jul 28 17:36:54.940804 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940606 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-home" (OuterVolumeSpecName: "home") pod "c9a2fe5d-fba4-4b35-ac30-c9133afbdede" (UID: "c9a2fe5d-fba4-4b35-ac30-c9133afbdede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:54.940804 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9a2fe5d-fba4-4b35-ac30-c9133afbdede" (UID: "c9a2fe5d-fba4-4b35-ac30-c9133afbdede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:54.940804 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9a2fe5d-fba4-4b35-ac30-c9133afbdede" (UID: "c9a2fe5d-fba4-4b35-ac30-c9133afbdede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:54.940804 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940737 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:36:54.940979 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940827 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:36:54.940979 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.940849 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:36:54.942642 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.942611 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-dshm" (OuterVolumeSpecName: "dshm") pod "c9a2fe5d-fba4-4b35-ac30-c9133afbdede" (UID: "c9a2fe5d-fba4-4b35-ac30-c9133afbdede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:54.943070 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.943047 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9a2fe5d-fba4-4b35-ac30-c9133afbdede" (UID: "c9a2fe5d-fba4-4b35-ac30-c9133afbdede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:36:54.943070 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:54.943060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-kube-api-access-4whwq" (OuterVolumeSpecName: "kube-api-access-4whwq") pod "c9a2fe5d-fba4-4b35-ac30-c9133afbdede" (UID: "c9a2fe5d-fba4-4b35-ac30-c9133afbdede"). InnerVolumeSpecName "kube-api-access-4whwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:36:55.041494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.041409 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:36:55.041494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.041436 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:36:55.041494 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.041448 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whwq\" (UniqueName: \"kubernetes.io/projected/c9a2fe5d-fba4-4b35-ac30-c9133afbdede-kube-api-access-4whwq\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:36:55.379119 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.379086 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" containerID="65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" exitCode=0 Jul 28 17:36:55.379316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.379135 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" Jul 28 17:36:55.379316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.379170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" event={"ID":"c9a2fe5d-fba4-4b35-ac30-c9133afbdede","Type":"ContainerDied","Data":"65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3"} Jul 28 17:36:55.379316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.379222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj" event={"ID":"c9a2fe5d-fba4-4b35-ac30-c9133afbdede","Type":"ContainerDied","Data":"d02d5b69789fed033eac4a8350d31726d097eb943217c382799aaa2571f44b48"} Jul 28 17:36:55.379316 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.379251 2564 scope.go:117] "RemoveContainer" containerID="65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" Jul 28 17:36:55.389020 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.389002 2564 scope.go:117] "RemoveContainer" containerID="65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" Jul 28 17:36:55.389335 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:36:55.389318 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3\": container with ID starting with 65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3 not found: ID does not exist" containerID="65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3" Jul 28 17:36:55.389382 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.389345 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3"} err="failed to get container status \"65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3\": rpc error: code = NotFound desc = could not find container \"65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3\": container with ID starting with 65ef3e56e14fd50ff81eb1340206d64999587baeb8a185dd185f34c5a34fd4d3 not found: ID does not exist" Jul 28 17:36:55.402855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.402820 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj"] Jul 28 17:36:55.408766 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:55.408733 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4mtcj"] Jul 28 17:36:57.376808 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:36:57.376751 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" path="/var/lib/kubelet/pods/c9a2fe5d-fba4-4b35-ac30-c9133afbdede/volumes" Jul 28 17:37:22.110697 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.110652 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j"] Jul 28 17:37:22.111245 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.111224 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" containerName="main" Jul 28 17:37:22.111317 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.111248 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" containerName="main" Jul 28 17:37:22.111385 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.111372 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a2fe5d-fba4-4b35-ac30-c9133afbdede" containerName="main" Jul 28 17:37:22.114241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.114218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.118575 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.118538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\"" Jul 28 17:37:22.118575 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.118560 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\"" Jul 28 17:37:22.118822 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.118580 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\"" Jul 28 17:37:22.118822 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.118538 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-mt4zr\"" Jul 28 17:37:22.124792 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.124750 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j"] Jul 28 17:37:22.174264 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.174221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.174264 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.174262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6fb\" (UniqueName: \"kubernetes.io/projected/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-kube-api-access-nz6fb\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.174491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.174289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.174491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.174387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.174491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.174429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.174595 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.174552 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-home\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276048 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.275949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-home\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276240 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276240 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6fb\" (UniqueName: \"kubernetes.io/projected/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-kube-api-access-nz6fb\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276240 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276240 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276240 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276523 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276480 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-home\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.276746 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.276723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.278647 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.278614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.278918 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.278899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.284304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.284266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6fb\" (UniqueName: \"kubernetes.io/projected/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-kube-api-access-nz6fb\") pod \"fc-concurrency-test-kserve-b5d44467c-mkh5j\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.425407 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.425309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:22.569270 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:22.569243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j"] Jul 28 17:37:22.571338 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:37:22.571305 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3928a33a_c9a4_4e93_bd6a_e04a6cab4226.slice/crio-017451264ceb245dc5326b25ae0bb8ac49049d9e10374947f8033e4c2d5d4916 WatchSource:0}: Error finding container 017451264ceb245dc5326b25ae0bb8ac49049d9e10374947f8033e4c2d5d4916: Status 404 returned error can't find the container with id 017451264ceb245dc5326b25ae0bb8ac49049d9e10374947f8033e4c2d5d4916 Jul 28 17:37:23.517331 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:23.517295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" event={"ID":"3928a33a-c9a4-4e93-bd6a-e04a6cab4226","Type":"ContainerStarted","Data":"51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad"} Jul 28 17:37:23.517712 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:23.517350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" event={"ID":"3928a33a-c9a4-4e93-bd6a-e04a6cab4226","Type":"ContainerStarted","Data":"017451264ceb245dc5326b25ae0bb8ac49049d9e10374947f8033e4c2d5d4916"} Jul 28 17:37:23.553606 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:23.553551 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" podStartSLOduration=1.5535312220000002 podStartE2EDuration="1.553531222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:23.550193002 +0000 UTC m=+876.702693179" watchObservedRunningTime="2026-07-28 17:37:23.553531222 +0000 UTC m=+876.706031398" Jul 28 17:37:24.425884 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:24.425841 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:32.426344 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:32.426302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:32.438925 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:32.438893 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:32.573422 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:32.573393 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:37:47.391419 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:47.391390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:37:47.392888 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:37:47.392868 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:38:01.171914 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.171879 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j"] Jul 28 17:38:01.181973 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:01.181937 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:38:01.181973 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:01.181973 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:01.181973 ip-10-0-131-29 kubenswrapper[2564]: > containerID="51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" cmd=["/bin/sleep","15"] Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.181988 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" message="" err=< Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:01.182001 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" podUID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" containerName="main" containerID="cri-o://51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" Jul 28 17:38:01.182295 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.182050 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" podUID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" containerName="main" containerID="cri-o://51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" gracePeriod=60 Jul 28 17:38:01.541401 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.541371 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:38:01.644194 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644157 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz6fb\" (UniqueName: \"kubernetes.io/projected/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-kube-api-access-nz6fb\") pod \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " Jul 28 17:38:01.644406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644224 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-model-cache\") pod \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " Jul 28 17:38:01.644406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-home\") pod \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " Jul 28 17:38:01.644406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644314 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tls-certs\") pod \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " Jul 28 17:38:01.644406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-dshm\") pod \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " Jul 28 17:38:01.644605 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644409 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tmp-dir\") pod \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\" (UID: \"3928a33a-c9a4-4e93-bd6a-e04a6cab4226\") " Jul 28 17:38:01.644605 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644532 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-model-cache" (OuterVolumeSpecName: "model-cache") pod "3928a33a-c9a4-4e93-bd6a-e04a6cab4226" (UID: "3928a33a-c9a4-4e93-bd6a-e04a6cab4226"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:01.644605 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644545 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-home" (OuterVolumeSpecName: "home") pod "3928a33a-c9a4-4e93-bd6a-e04a6cab4226" (UID: "3928a33a-c9a4-4e93-bd6a-e04a6cab4226"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:01.644798 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644735 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3928a33a-c9a4-4e93-bd6a-e04a6cab4226" (UID: "3928a33a-c9a4-4e93-bd6a-e04a6cab4226"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:01.644798 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644745 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:01.644921 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.644816 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:01.646597 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.646543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-dshm" (OuterVolumeSpecName: "dshm") pod "3928a33a-c9a4-4e93-bd6a-e04a6cab4226" (UID: "3928a33a-c9a4-4e93-bd6a-e04a6cab4226"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:01.646719 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.646620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-kube-api-access-nz6fb" (OuterVolumeSpecName: "kube-api-access-nz6fb") pod "3928a33a-c9a4-4e93-bd6a-e04a6cab4226" (UID: "3928a33a-c9a4-4e93-bd6a-e04a6cab4226"). InnerVolumeSpecName "kube-api-access-nz6fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:38:01.646719 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.646620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3928a33a-c9a4-4e93-bd6a-e04a6cab4226" (UID: "3928a33a-c9a4-4e93-bd6a-e04a6cab4226"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:38:01.710807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.710697 2564 generic.go:356] "Generic (PLEG): container finished" podID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" containerID="51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" exitCode=0 Jul 28 17:38:01.710807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.710740 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" Jul 28 17:38:01.711006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.710794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" event={"ID":"3928a33a-c9a4-4e93-bd6a-e04a6cab4226","Type":"ContainerDied","Data":"51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad"} Jul 28 17:38:01.711006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.710863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j" event={"ID":"3928a33a-c9a4-4e93-bd6a-e04a6cab4226","Type":"ContainerDied","Data":"017451264ceb245dc5326b25ae0bb8ac49049d9e10374947f8033e4c2d5d4916"} Jul 28 17:38:01.711006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.710888 2564 scope.go:117] "RemoveContainer" containerID="51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" Jul 28 17:38:01.720650 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.720630 2564 scope.go:117] "RemoveContainer" containerID="51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" Jul 28 17:38:01.720967 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:01.720947 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad\": container with ID starting with 51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad not found: ID does not exist" containerID="51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad" Jul 28 17:38:01.721041 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.720978 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad"} err="failed to get container status \"51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad\": rpc error: code = NotFound desc = could not find container \"51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad\": container with ID starting with 51d2c4e36ee52312a51e3a18190b63cdc20f29980d582d0d5967ccff6994b0ad not found: ID does not exist" Jul 28 17:38:01.734213 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.734177 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j"] Jul 28 17:38:01.738569 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.738539 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mkh5j"] Jul 28 17:38:01.746046 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.746021 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:01.746168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.746069 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:01.746168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.746078 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6fb\" (UniqueName: \"kubernetes.io/projected/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-kube-api-access-nz6fb\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:01.746168 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:01.746088 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a33a-c9a4-4e93-bd6a-e04a6cab4226-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:03.376950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:03.376921 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" path="/var/lib/kubelet/pods/3928a33a-c9a4-4e93-bd6a-e04a6cab4226/volumes" Jul 28 17:38:29.493318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.493261 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn"] Jul 28 17:38:29.493716 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.493680 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" containerName="main" Jul 28 17:38:29.493716 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.493691 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" containerName="main" Jul 28 17:38:29.493826 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.493784 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3928a33a-c9a4-4e93-bd6a-e04a6cab4226" containerName="main" Jul 28 17:38:29.496638 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.496621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.501080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.501038 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-rtwq9\"" Jul 28 17:38:29.501080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.501079 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\"" Jul 28 17:38:29.501350 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.501048 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\"" Jul 28 17:38:29.501404 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.501375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Jul 28 17:38:29.507203 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.507174 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn"] Jul 28 17:38:29.598621 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.598582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.598621 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.598634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.598975 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.598672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.598975 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.598691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.598975 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.598707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l755z\" (UniqueName: \"kubernetes.io/projected/58e9c324-9644-4404-95ba-8da3479f0a6b-kube-api-access-l755z\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.598975 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.598810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700248 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700466 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700466 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700466 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700466 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700466 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755z\" (UniqueName: \"kubernetes.io/projected/58e9c324-9644-4404-95ba-8da3479f0a6b-kube-api-access-l755z\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700856 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700856 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.700957 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.700883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.702652 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.702626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.702885 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.702867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.710553 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.710515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l755z\" (UniqueName: \"kubernetes.io/projected/58e9c324-9644-4404-95ba-8da3479f0a6b-kube-api-access-l755z\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.809192 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.809096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:29.963786 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:29.963743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn"] Jul 28 17:38:29.966492 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:38:29.966449 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e9c324_9644_4404_95ba_8da3479f0a6b.slice/crio-b662d1c03a68cbdbe630ea69f67ca8bc961ed4389205edded0f2899f8a619ec3 WatchSource:0}: Error finding container b662d1c03a68cbdbe630ea69f67ca8bc961ed4389205edded0f2899f8a619ec3: Status 404 returned error can't find the container with id b662d1c03a68cbdbe630ea69f67ca8bc961ed4389205edded0f2899f8a619ec3 Jul 28 17:38:30.863875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:30.863841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" event={"ID":"58e9c324-9644-4404-95ba-8da3479f0a6b","Type":"ContainerStarted","Data":"dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7"} Jul 28 17:38:30.864267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:30.863899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" event={"ID":"58e9c324-9644-4404-95ba-8da3479f0a6b","Type":"ContainerStarted","Data":"b662d1c03a68cbdbe630ea69f67ca8bc961ed4389205edded0f2899f8a619ec3"} Jul 28 17:38:30.883972 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:30.883917 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" podStartSLOduration=1.883898008 podStartE2EDuration="1.883898008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:30.88274899 +0000 UTC m=+944.035249167" watchObservedRunningTime="2026-07-28 17:38:30.883898008 +0000 UTC m=+944.036398185" Jul 28 17:38:31.809484 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:31.809441 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:31.913571 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:31.913537 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn"] Jul 28 17:38:32.021707 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:32.021668 2564 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found Jul 28 17:38:32.021930 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:32.021749 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs podName:58e9c324-9644-4404-95ba-8da3479f0a6b nodeName:}" failed. No retries permitted until 2026-07-28 17:38:32.521732217 +0000 UTC m=+945.674232376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found Jul 28 17:38:32.526983 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:32.526934 2564 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found Jul 28 17:38:32.527193 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:32.527013 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs podName:58e9c324-9644-4404-95ba-8da3479f0a6b nodeName:}" failed. No retries permitted until 2026-07-28 17:38:33.526996344 +0000 UTC m=+946.679496504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found Jul 28 17:38:33.536949 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:33.536905 2564 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found Jul 28 17:38:33.537373 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:33.536992 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs podName:58e9c324-9644-4404-95ba-8da3479f0a6b nodeName:}" failed. No retries permitted until 2026-07-28 17:38:35.536977095 +0000 UTC m=+948.689477249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found Jul 28 17:38:33.895409 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:33.895371 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:38:33.895409 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:33.895409 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:33.895409 ip-10-0-131-29 kubenswrapper[2564]: > containerID="dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" cmd=["/bin/sleep","15"] Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:33.895427 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" message="" err=< Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:33.895440 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" podUID="58e9c324-9644-4404-95ba-8da3479f0a6b" containerName="main" containerID="cri-o://dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" Jul 28 17:38:33.895707 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:33.895490 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" podUID="58e9c324-9644-4404-95ba-8da3479f0a6b" containerName="main" containerID="cri-o://dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" gracePeriod=60 Jul 28 17:38:34.156485 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.156409 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:34.243500 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-tmp-dir\") pod \"58e9c324-9644-4404-95ba-8da3479f0a6b\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " Jul 28 17:38:34.243685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs\") pod \"58e9c324-9644-4404-95ba-8da3479f0a6b\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " Jul 28 17:38:34.243685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243594 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-model-cache\") pod \"58e9c324-9644-4404-95ba-8da3479f0a6b\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " Jul 28 17:38:34.243685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-dshm\") pod \"58e9c324-9644-4404-95ba-8da3479f0a6b\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " Jul 28 17:38:34.243685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-home\") pod \"58e9c324-9644-4404-95ba-8da3479f0a6b\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " Jul 28 17:38:34.243685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243680 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l755z\" (UniqueName: \"kubernetes.io/projected/58e9c324-9644-4404-95ba-8da3479f0a6b-kube-api-access-l755z\") pod \"58e9c324-9644-4404-95ba-8da3479f0a6b\" (UID: \"58e9c324-9644-4404-95ba-8da3479f0a6b\") " Jul 28 17:38:34.243961 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243838 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58e9c324-9644-4404-95ba-8da3479f0a6b" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:34.243961 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243875 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "58e9c324-9644-4404-95ba-8da3479f0a6b" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:34.244060 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243978 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:34.244060 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.243992 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:34.244166 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.244053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-home" (OuterVolumeSpecName: "home") pod "58e9c324-9644-4404-95ba-8da3479f0a6b" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:34.246022 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.245985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-dshm" (OuterVolumeSpecName: "dshm") pod "58e9c324-9644-4404-95ba-8da3479f0a6b" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:34.246144 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.246059 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58e9c324-9644-4404-95ba-8da3479f0a6b" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:38:34.246144 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.246083 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e9c324-9644-4404-95ba-8da3479f0a6b-kube-api-access-l755z" (OuterVolumeSpecName: "kube-api-access-l755z") pod "58e9c324-9644-4404-95ba-8da3479f0a6b" (UID: "58e9c324-9644-4404-95ba-8da3479f0a6b"). InnerVolumeSpecName "kube-api-access-l755z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:38:34.345325 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.345284 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l755z\" (UniqueName: \"kubernetes.io/projected/58e9c324-9644-4404-95ba-8da3479f0a6b-kube-api-access-l755z\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:34.345325 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.345317 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e9c324-9644-4404-95ba-8da3479f0a6b-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:34.345325 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.345330 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:34.345325 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.345338 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e9c324-9644-4404-95ba-8da3479f0a6b-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:34.892380 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.892344 2564 generic.go:356] "Generic (PLEG): container finished" podID="58e9c324-9644-4404-95ba-8da3479f0a6b" containerID="dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" exitCode=0 Jul 28 17:38:34.892855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.892388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" Jul 28 17:38:34.892855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.892425 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" event={"ID":"58e9c324-9644-4404-95ba-8da3479f0a6b","Type":"ContainerDied","Data":"dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7"} Jul 28 17:38:34.892855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.892479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn" event={"ID":"58e9c324-9644-4404-95ba-8da3479f0a6b","Type":"ContainerDied","Data":"b662d1c03a68cbdbe630ea69f67ca8bc961ed4389205edded0f2899f8a619ec3"} Jul 28 17:38:34.892855 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.892502 2564 scope.go:117] "RemoveContainer" containerID="dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" Jul 28 17:38:34.901947 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.901728 2564 scope.go:117] "RemoveContainer" containerID="dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" Jul 28 17:38:34.902058 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:34.902038 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7\": container with ID starting with dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7 not found: ID does not exist" containerID="dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7" Jul 28 17:38:34.902122 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.902092 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7"} err="failed to get container status \"dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7\": rpc error: code = NotFound desc = could not find container \"dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7\": container with ID starting with dcb4565aee02d19edd43a8659fade931e51b84ac0541ea753f3cd56266edc8b7 not found: ID does not exist" Jul 28 17:38:34.914206 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.914170 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn"] Jul 28 17:38:34.920643 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:34.920619 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dfphn"] Jul 28 17:38:35.376714 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:35.376683 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e9c324-9644-4404-95ba-8da3479f0a6b" path="/var/lib/kubelet/pods/58e9c324-9644-4404-95ba-8da3479f0a6b/volumes" Jul 28 17:38:46.488860 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.488829 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w"] Jul 28 17:38:46.489255 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.489239 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e9c324-9644-4404-95ba-8da3479f0a6b" containerName="main" Jul 28 17:38:46.489255 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.489257 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e9c324-9644-4404-95ba-8da3479f0a6b" containerName="main" Jul 28 17:38:46.489350 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.489330 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e9c324-9644-4404-95ba-8da3479f0a6b" containerName="main" Jul 28 17:38:46.492462 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.492443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.495378 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.495354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\"" Jul 28 17:38:46.496696 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.496675 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-xlg75\"" Jul 28 17:38:46.496868 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.496852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Jul 28 17:38:46.496933 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.496869 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\"" Jul 28 17:38:46.501909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.501377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w"] Jul 28 17:38:46.556343 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.556304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.556514 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.556367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.556514 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.556388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.556514 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.556409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.556514 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.556427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rb5\" (UniqueName: \"kubernetes.io/projected/40ab4a53-e30b-41f4-8f36-9a7b45a65625-kube-api-access-52rb5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.556514 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.556471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657241 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657458 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657458 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657458 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657458 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rb5\" (UniqueName: \"kubernetes.io/projected/40ab4a53-e30b-41f4-8f36-9a7b45a65625-kube-api-access-52rb5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657458 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657697 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.657905 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.657886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.659634 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.659613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.660068 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.660049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.666066 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.666035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rb5\" (UniqueName: \"kubernetes.io/projected/40ab4a53-e30b-41f4-8f36-9a7b45a65625-kube-api-access-52rb5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.805715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.805623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:46.948024 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:46.947998 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w"] Jul 28 17:38:46.950554 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:38:46.950510 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ab4a53_e30b_41f4_8f36_9a7b45a65625.slice/crio-0cec62f0297750b2da2ae9936d2f296977540a8425b0657f0dd8e5cc5455686f WatchSource:0}: Error finding container 0cec62f0297750b2da2ae9936d2f296977540a8425b0657f0dd8e5cc5455686f: Status 404 returned error can't find the container with id 0cec62f0297750b2da2ae9936d2f296977540a8425b0657f0dd8e5cc5455686f Jul 28 17:38:47.960811 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:47.960751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" event={"ID":"40ab4a53-e30b-41f4-8f36-9a7b45a65625","Type":"ContainerStarted","Data":"5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076"} Jul 28 17:38:47.961192 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:47.960864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" event={"ID":"40ab4a53-e30b-41f4-8f36-9a7b45a65625","Type":"ContainerStarted","Data":"0cec62f0297750b2da2ae9936d2f296977540a8425b0657f0dd8e5cc5455686f"} Jul 28 17:38:47.978438 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:47.978387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" podStartSLOduration=1.978371515 podStartE2EDuration="1.978371515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:47.976836291 +0000 UTC m=+961.129336493" watchObservedRunningTime="2026-07-28 17:38:47.978371515 +0000 UTC m=+961.130871691" Jul 28 17:38:48.806780 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:48.806712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:48.957684 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:48.957650 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w"] Jul 28 17:38:48.975467 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:48.975434 2564 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 28 17:38:48.975902 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:48.975509 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs podName:40ab4a53-e30b-41f4-8f36-9a7b45a65625 nodeName:}" failed. No retries permitted until 2026-07-28 17:38:49.475489302 +0000 UTC m=+962.627989457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 28 17:38:49.479887 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:49.479802 2564 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 28 17:38:49.480022 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:49.479887 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs podName:40ab4a53-e30b-41f4-8f36-9a7b45a65625 nodeName:}" failed. No retries permitted until 2026-07-28 17:38:50.479871396 +0000 UTC m=+963.632371551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found Jul 28 17:38:49.981794 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:49.981738 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:38:49.981794 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:49.981794 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:49.981794 ip-10-0-131-29 kubenswrapper[2564]: > containerID="5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" cmd=["/bin/sleep","15"] Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:49.981815 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" message="" err=< Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:49.981828 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" podUID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" containerName="main" containerID="cri-o://5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" Jul 28 17:38:49.982343 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:49.981881 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" podUID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" containerName="main" containerID="cri-o://5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" gracePeriod=60 Jul 28 17:38:50.238533 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.238450 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:50.286594 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-dshm\") pod \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " Jul 28 17:38:50.286789 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs\") pod \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " Jul 28 17:38:50.286789 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286651 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tmp-dir\") pod \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " Jul 28 17:38:50.286789 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-home\") pod \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " Jul 28 17:38:50.286789 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286746 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rb5\" (UniqueName: \"kubernetes.io/projected/40ab4a53-e30b-41f4-8f36-9a7b45a65625-kube-api-access-52rb5\") pod \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " Jul 28 17:38:50.287040 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286853 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-model-cache\") pod \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\" (UID: \"40ab4a53-e30b-41f4-8f36-9a7b45a65625\") " Jul 28 17:38:50.287040 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286977 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40ab4a53-e30b-41f4-8f36-9a7b45a65625" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:50.287040 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.286996 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-home" (OuterVolumeSpecName: "home") pod "40ab4a53-e30b-41f4-8f36-9a7b45a65625" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:50.287194 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.287154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-model-cache" (OuterVolumeSpecName: "model-cache") pod "40ab4a53-e30b-41f4-8f36-9a7b45a65625" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:50.287249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.287198 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:50.287249 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.287212 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:50.288955 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.288931 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-dshm" (OuterVolumeSpecName: "dshm") pod "40ab4a53-e30b-41f4-8f36-9a7b45a65625" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:38:50.289052 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.288973 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40ab4a53-e30b-41f4-8f36-9a7b45a65625" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:38:50.289052 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.289027 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ab4a53-e30b-41f4-8f36-9a7b45a65625-kube-api-access-52rb5" (OuterVolumeSpecName: "kube-api-access-52rb5") pod "40ab4a53-e30b-41f4-8f36-9a7b45a65625" (UID: "40ab4a53-e30b-41f4-8f36-9a7b45a65625"). InnerVolumeSpecName "kube-api-access-52rb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:38:50.388029 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.387994 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:50.388029 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.388020 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ab4a53-e30b-41f4-8f36-9a7b45a65625-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:50.388029 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.388033 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rb5\" (UniqueName: \"kubernetes.io/projected/40ab4a53-e30b-41f4-8f36-9a7b45a65625-kube-api-access-52rb5\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:50.388258 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.388042 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ab4a53-e30b-41f4-8f36-9a7b45a65625-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:38:50.979567 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.979536 2564 generic.go:356] "Generic (PLEG): container finished" podID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" containerID="5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" exitCode=0 Jul 28 17:38:50.979781 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.979583 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" Jul 28 17:38:50.979781 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.979629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" event={"ID":"40ab4a53-e30b-41f4-8f36-9a7b45a65625","Type":"ContainerDied","Data":"5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076"} Jul 28 17:38:50.979781 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.979678 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w" event={"ID":"40ab4a53-e30b-41f4-8f36-9a7b45a65625","Type":"ContainerDied","Data":"0cec62f0297750b2da2ae9936d2f296977540a8425b0657f0dd8e5cc5455686f"} Jul 28 17:38:50.979781 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.979702 2564 scope.go:117] "RemoveContainer" containerID="5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" Jul 28 17:38:50.989275 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.989094 2564 scope.go:117] "RemoveContainer" containerID="5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" Jul 28 17:38:50.989491 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:38:50.989427 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076\": container with ID starting with 5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076 not found: ID does not exist" containerID="5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076" Jul 28 17:38:50.989491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:50.989449 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076"} err="failed to get container status \"5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076\": rpc error: code = NotFound desc = could not find container \"5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076\": container with ID starting with 5d17b4bd01135832c1d8cda23110d77ed5111e7e40ba87a9355f6e3fb267a076 not found: ID does not exist" Jul 28 17:38:51.002698 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:51.002664 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w"] Jul 28 17:38:51.004393 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:51.004363 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfclf7w"] Jul 28 17:38:51.377533 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:38:51.377495 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" path="/var/lib/kubelet/pods/40ab4a53-e30b-41f4-8f36-9a7b45a65625/volumes" Jul 28 17:39:02.494636 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.494600 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc"] Jul 28 17:39:02.495781 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.495740 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" containerName="main" Jul 28 17:39:02.495909 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.495897 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" containerName="main" Jul 28 17:39:02.496115 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.496103 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ab4a53-e30b-41f4-8f36-9a7b45a65625" containerName="main" Jul 28 17:39:02.501356 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.501337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.505376 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.505344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\"" Jul 28 17:39:02.505708 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.505681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-nt7w8\"" Jul 28 17:39:02.506099 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.506078 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\"" Jul 28 17:39:02.506281 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.506243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\"" Jul 28 17:39:02.506281 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.506268 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\"" Jul 28 17:39:02.512373 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.512348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc"] Jul 28 17:39:02.598685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-dshm\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.598685 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-kserve-provision-location\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.598917 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx7pw\" (UniqueName: \"kubernetes.io/projected/ffc06205-40bf-49fa-ad83-2e5327222014-kube-api-access-gx7pw\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.598917 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-home\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.598917 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-tmp-dir\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.598917 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-model-cache\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.599089 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.598938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc06205-40bf-49fa-ad83-2e5327222014-tls-certs\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699595 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-dshm\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699595 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-kserve-provision-location\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx7pw\" (UniqueName: \"kubernetes.io/projected/ffc06205-40bf-49fa-ad83-2e5327222014-kube-api-access-gx7pw\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-home\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-tmp-dir\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-model-cache\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.699875 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.699720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc06205-40bf-49fa-ad83-2e5327222014-tls-certs\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.700150 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.700086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-kserve-provision-location\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.700204 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.700151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-model-cache\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.700255 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.700219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-tmp-dir\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.700309 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.700253 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-home\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.702046 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.702025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-dshm\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.702138 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.702107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc06205-40bf-49fa-ad83-2e5327222014-tls-certs\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.708006 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.707988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx7pw\" (UniqueName: \"kubernetes.io/projected/ffc06205-40bf-49fa-ad83-2e5327222014-kube-api-access-gx7pw\") pod \"auth-enabled-test-kserve-777b6c898c-p2bhc\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.814738 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.814644 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:02.961788 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:02.961743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc"] Jul 28 17:39:02.963519 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:39:02.963492 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc06205_40bf_49fa_ad83_2e5327222014.slice/crio-1efc5664fd9f66cce98cf422a8a4ca3dfe59b64074f6346bfc095e636b17c3ca WatchSource:0}: Error finding container 1efc5664fd9f66cce98cf422a8a4ca3dfe59b64074f6346bfc095e636b17c3ca: Status 404 returned error can't find the container with id 1efc5664fd9f66cce98cf422a8a4ca3dfe59b64074f6346bfc095e636b17c3ca Jul 28 17:39:03.044312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:03.044268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" event={"ID":"ffc06205-40bf-49fa-ad83-2e5327222014","Type":"ContainerStarted","Data":"1efc5664fd9f66cce98cf422a8a4ca3dfe59b64074f6346bfc095e636b17c3ca"} Jul 28 17:39:07.069998 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:07.069961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" event={"ID":"ffc06205-40bf-49fa-ad83-2e5327222014","Type":"ContainerStarted","Data":"1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442"} Jul 28 17:39:17.127365 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:17.127330 2564 generic.go:356] "Generic (PLEG): container finished" podID="ffc06205-40bf-49fa-ad83-2e5327222014" containerID="1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442" exitCode=0 Jul 28 17:39:17.127751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:17.127401 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" event={"ID":"ffc06205-40bf-49fa-ad83-2e5327222014","Type":"ContainerDied","Data":"1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442"} Jul 28 17:39:48.318680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:48.318642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" event={"ID":"ffc06205-40bf-49fa-ad83-2e5327222014","Type":"ContainerStarted","Data":"30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf"} Jul 28 17:39:48.346222 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:48.346155 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podStartSLOduration=2.820691615 podStartE2EDuration="46.346133599s" totalImagesPullingTime="33.944308839s" totalInitContainerRuntime="9.581133145s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:48.342594825 +0000 UTC m=+1021.495095003" watchObservedRunningTime="2026-07-28 17:39:48.346133599 +0000 UTC m=+1021.498633780" Jul 28 17:39:48.815051 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:48.815015 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:52.815711 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:52.815664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:39:52.817103 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:39:52.817068 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:40:01.201506 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.201458 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"] Jul 28 17:40:01.646347 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.646303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.649391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.649359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\"" Jul 28 17:40:01.650988 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.650962 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\"" Jul 28 17:40:01.651223 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.651051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\"" Jul 28 17:40:01.651366 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.651075 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-m4fbq\"" Jul 28 17:40:01.651434 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.651096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\"" Jul 28 17:40:01.651493 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.651481 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rgfwt\"" Jul 28 17:40:01.655517 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.655482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"] Jul 28 17:40:01.778667 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.778628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.778874 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.778778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.778874 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.778859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.779014 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.778971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.779076 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.779057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.779138 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.779096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxrh\" (UniqueName: \"kubernetes.io/projected/fe34ad27-a690-4777-ae5f-fa48b014c658-kube-api-access-zlxrh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.779195 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.779132 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880427 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880427 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxrh\" (UniqueName: \"kubernetes.io/projected/fe34ad27-a690-4777-ae5f-fa48b014c658-kube-api-access-zlxrh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880902 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880902 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.880902 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880870 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.881070 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.880917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.881128 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.881066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.881128 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.881089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.883162 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.883137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.883341 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.883175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.890315 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.890273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxrh\" (UniqueName: \"kubernetes.io/projected/fe34ad27-a690-4777-ae5f-fa48b014c658-kube-api-access-zlxrh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:01.959732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:01.959647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:40:02.152165 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:02.152135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"] Jul 28 17:40:02.154037 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:40:02.153995 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe34ad27_a690_4777_ae5f_fa48b014c658.slice/crio-d5765aff3bcda8313b1e447e3bcdea76b8fe6cfbd1f8b78e3426a25dc1a9ad84 WatchSource:0}: Error finding container d5765aff3bcda8313b1e447e3bcdea76b8fe6cfbd1f8b78e3426a25dc1a9ad84: Status 404 returned error can't find the container with id d5765aff3bcda8313b1e447e3bcdea76b8fe6cfbd1f8b78e3426a25dc1a9ad84 Jul 28 17:40:02.395795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:02.395738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe34ad27-a690-4777-ae5f-fa48b014c658","Type":"ContainerStarted","Data":"dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5"} Jul 28 17:40:02.396295 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:02.395822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe34ad27-a690-4777-ae5f-fa48b014c658","Type":"ContainerStarted","Data":"d5765aff3bcda8313b1e447e3bcdea76b8fe6cfbd1f8b78e3426a25dc1a9ad84"} Jul 28 17:40:02.815960 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:02.815907 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:40:12.460001 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:12.459956 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerID="dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5" exitCode=0 Jul 28 17:40:12.460001 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:12.459992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe34ad27-a690-4777-ae5f-fa48b014c658","Type":"ContainerDied","Data":"dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5"} Jul 28 17:40:12.815571 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:12.815461 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:40:13.468716 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:13.468677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe34ad27-a690-4777-ae5f-fa48b014c658","Type":"ContainerStarted","Data":"0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a"} Jul 28 17:40:13.496479 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:13.496402 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.2755167419999998 podStartE2EDuration="12.496380566s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.220863824s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:13.492642529 +0000 UTC m=+1046.645142734" watchObservedRunningTime="2026-07-28 17:40:13.496380566 +0000 UTC m=+1046.648880746" Jul 28 17:40:22.815541 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:22.815485 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:40:32.816106 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:32.816034 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:40:42.815170 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:42.815116 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:40:52.815430 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:40:52.815379 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:41:02.816135 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:41:02.816083 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:41:12.815400 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:41:12.815350 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:41:22.815470 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:41:22.815429 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:41:32.816185 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:41:32.816130 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:41:42.815245 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:41:42.815197 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:41:52.815680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:41:52.815632 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:42:02.816004 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:02.815955 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:42:12.816073 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:12.815984 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:42:22.815394 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:22.815344 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Jul 28 17:42:32.825440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:32.825400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:42:32.833716 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:32.833689 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:42:37.796141 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:37.796098 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc"] Jul 28 17:42:47.373859 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:47.373823 2564 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rgfwt\" not found" Jul 28 17:42:47.413524 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:47.413488 2564 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:47.413730 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:47.413620 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs podName:fe34ad27-a690-4777-ae5f-fa48b014c658 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:47.913596224 +0000 UTC m=+1201.066096381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:47.426114 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:47.426077 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:42:47.428137 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:47.428112 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:42:47.918779 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:47.918705 2564 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:47.918976 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:47.918869 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs podName:fe34ad27-a690-4777-ae5f-fa48b014c658 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:48.918851927 +0000 UTC m=+1202.071352082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:48.927419 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:48.927377 2564 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:48.927856 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:48.927463 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs podName:fe34ad27-a690-4777-ae5f-fa48b014c658 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:50.927447228 +0000 UTC m=+1204.079947384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:50.947136 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:50.947099 2564 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:50.947595 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:50.947170 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs podName:fe34ad27-a690-4777-ae5f-fa48b014c658 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:54.947155709 +0000 UTC m=+1208.099655864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:52.819109 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:52.819069 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" containerID="cri-o://30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf" gracePeriod=45 Jul 28 17:42:53.579814 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.579787 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:42:53.673513 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673415 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc06205-40bf-49fa-ad83-2e5327222014-tls-certs\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.673513 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-kserve-provision-location\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.673513 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673510 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-dshm\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.673829 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673527 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-home\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.673829 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx7pw\" (UniqueName: \"kubernetes.io/projected/ffc06205-40bf-49fa-ad83-2e5327222014-kube-api-access-gx7pw\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.673829 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-model-cache\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.673829 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-tmp-dir\") pod \"ffc06205-40bf-49fa-ad83-2e5327222014\" (UID: \"ffc06205-40bf-49fa-ad83-2e5327222014\") " Jul 28 17:42:53.674036 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.673936 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:53.674326 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.674300 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-home" (OuterVolumeSpecName: "home") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:53.675904 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.675874 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-dshm" (OuterVolumeSpecName: "dshm") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:53.676183 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.676163 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc06205-40bf-49fa-ad83-2e5327222014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:42:53.676387 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.676368 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc06205-40bf-49fa-ad83-2e5327222014-kube-api-access-gx7pw" (OuterVolumeSpecName: "kube-api-access-gx7pw") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "kube-api-access-gx7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:42:53.686715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.686675 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:53.730698 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.730650 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffc06205-40bf-49fa-ad83-2e5327222014" (UID: "ffc06205-40bf-49fa-ad83-2e5327222014"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:53.774539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774500 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:53.774539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774534 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:53.774539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774542 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc06205-40bf-49fa-ad83-2e5327222014-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:53.774825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774552 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:53.774825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774562 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:53.774825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774571 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc06205-40bf-49fa-ad83-2e5327222014-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:53.774825 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:53.774579 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx7pw\" (UniqueName: \"kubernetes.io/projected/ffc06205-40bf-49fa-ad83-2e5327222014-kube-api-access-gx7pw\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:42:54.352715 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.352671 2564 generic.go:356] "Generic (PLEG): container finished" podID="ffc06205-40bf-49fa-ad83-2e5327222014" containerID="30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf" exitCode=0 Jul 28 17:42:54.353157 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.352779 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" Jul 28 17:42:54.353157 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.352779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" event={"ID":"ffc06205-40bf-49fa-ad83-2e5327222014","Type":"ContainerDied","Data":"30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf"} Jul 28 17:42:54.353157 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.352866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc" event={"ID":"ffc06205-40bf-49fa-ad83-2e5327222014","Type":"ContainerDied","Data":"1efc5664fd9f66cce98cf422a8a4ca3dfe59b64074f6346bfc095e636b17c3ca"} Jul 28 17:42:54.353157 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.352890 2564 scope.go:117] "RemoveContainer" containerID="30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf" Jul 28 17:42:54.362881 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.362850 2564 scope.go:117] "RemoveContainer" containerID="1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442" Jul 28 17:42:54.374408 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.374386 2564 scope.go:117] "RemoveContainer" containerID="30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf" Jul 28 17:42:54.374716 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:54.374693 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf\": container with ID starting with 30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf not found: ID does not exist" containerID="30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf" Jul 28 17:42:54.374776 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.374727 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf"} err="failed to get container status \"30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf\": rpc error: code = NotFound desc = could not find container \"30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf\": container with ID starting with 30ecc0fd6dcb845ff83b256f87a1e2fe86d3e82a3e3b4c094e6691350096b9cf not found: ID does not exist" Jul 28 17:42:54.374776 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.374746 2564 scope.go:117] "RemoveContainer" containerID="1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442" Jul 28 17:42:54.375042 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:54.375025 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442\": container with ID starting with 1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442 not found: ID does not exist" containerID="1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442" Jul 28 17:42:54.375083 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.375049 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442"} err="failed to get container status \"1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442\": rpc error: code = NotFound desc = could not find container \"1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442\": container with ID starting with 1486d869b7045ac9949743f0e28a19f4c920509d8d32e1c471049a6759aeb442 not found: ID does not exist" Jul 28 17:42:54.380981 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.380946 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc"] Jul 28 17:42:54.385263 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:54.385234 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-p2bhc"] Jul 28 17:42:54.987687 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:54.987654 2564 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:54.987877 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:42:54.987738 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs podName:fe34ad27-a690-4777-ae5f-fa48b014c658 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:02.987722295 +0000 UTC m=+1216.140222449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:42:55.377252 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:42:55.377212 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" path="/var/lib/kubelet/pods/ffc06205-40bf-49fa-ad83-2e5327222014/volumes" Jul 28 17:43:00.030188 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:00.030149 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"] Jul 28 17:43:03.065984 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:43:03.065937 2564 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:43:03.066409 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:43:03.066023 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs podName:fe34ad27-a690-4777-ae5f-fa48b014c658 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:19.066007919 +0000 UTC m=+1232.218508074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found Jul 28 17:43:15.052524 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.052460 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerName="main" containerID="cri-o://0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a" gracePeriod=45 Jul 28 17:43:15.810879 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.810850 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:43:15.993003 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.992961 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-tmp-dir\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993003 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993010 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-kserve-provision-location\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993245 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993136 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993245 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993167 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-home\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993245 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993195 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-dshm\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993245 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993218 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-model-cache\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlxrh\" (UniqueName: \"kubernetes.io/projected/fe34ad27-a690-4777-ae5f-fa48b014c658-kube-api-access-zlxrh\") pod \"fe34ad27-a690-4777-ae5f-fa48b014c658\" (UID: \"fe34ad27-a690-4777-ae5f-fa48b014c658\") " Jul 28 17:43:15.993539 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993498 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:15.993911 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.993881 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-home" (OuterVolumeSpecName: "home") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:15.995534 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.995501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:43:15.995687 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.995668 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe34ad27-a690-4777-ae5f-fa48b014c658-kube-api-access-zlxrh" (OuterVolumeSpecName: "kube-api-access-zlxrh") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "kube-api-access-zlxrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:43:15.995738 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:15.995708 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-dshm" (OuterVolumeSpecName: "dshm") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:16.005970 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.005934 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:16.061476 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.061436 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe34ad27-a690-4777-ae5f-fa48b014c658" (UID: "fe34ad27-a690-4777-ae5f-fa48b014c658"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:16.094014 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.093976 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxrh\" (UniqueName: \"kubernetes.io/projected/fe34ad27-a690-4777-ae5f-fa48b014c658-kube-api-access-zlxrh\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.094014 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.094011 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.094014 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.094022 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.094262 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.094034 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34ad27-a690-4777-ae5f-fa48b014c658-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.094262 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.094044 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.094262 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.094052 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.094262 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.094060 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34ad27-a690-4777-ae5f-fa48b014c658-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.473878 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.473845 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerID="0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a" exitCode=0 Jul 28 17:43:16.474085 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.473904 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" Jul 28 17:43:16.474085 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.473931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe34ad27-a690-4777-ae5f-fa48b014c658","Type":"ContainerDied","Data":"0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a"} Jul 28 17:43:16.474085 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.473984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe34ad27-a690-4777-ae5f-fa48b014c658","Type":"ContainerDied","Data":"d5765aff3bcda8313b1e447e3bcdea76b8fe6cfbd1f8b78e3426a25dc1a9ad84"} Jul 28 17:43:16.474085 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.474008 2564 scope.go:117] "RemoveContainer" containerID="0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a" Jul 28 17:43:16.483892 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.483871 2564 scope.go:117] "RemoveContainer" containerID="dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5" Jul 28 17:43:16.492535 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.492507 2564 scope.go:117] "RemoveContainer" containerID="0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a" Jul 28 17:43:16.492832 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:43:16.492811 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a\": container with ID starting with 0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a not found: ID does not exist" containerID="0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a" Jul 28 17:43:16.492893 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.492842 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a"} err="failed to get container status \"0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a\": rpc error: code = NotFound desc = could not find container \"0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a\": container with ID starting with 0f4a44a6c1175fd6234a1f55ab9378f712fa16d87dbe3b4232f844214d8afa8a not found: ID does not exist" Jul 28 17:43:16.492893 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.492861 2564 scope.go:117] "RemoveContainer" containerID="dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5" Jul 28 17:43:16.496060 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:43:16.493290 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5\": container with ID starting with dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5 not found: ID does not exist" containerID="dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5" Jul 28 17:43:16.496060 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.493321 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5"} err="failed to get container status \"dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5\": rpc error: code = NotFound desc = could not find container \"dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5\": container with ID starting with dfa40ed02480dcfbccb7b88046a47f65eb0546a54bf8ff856c70ff57938a33c5 not found: ID does not exist" Jul 28 17:43:16.500139 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.500113 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"] Jul 28 17:43:16.507866 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:16.507836 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"] Jul 28 17:43:17.378597 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:17.378557 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" path="/var/lib/kubelet/pods/fe34ad27-a690-4777-ae5f-fa48b014c658/volumes" Jul 28 17:43:31.515977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.515945 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx"] Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516327 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerName="main" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516339 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerName="main" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516358 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516364 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516371 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerName="storage-initializer" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516378 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerName="storage-initializer" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516389 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="storage-initializer" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516397 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="storage-initializer" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516502 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc06205-40bf-49fa-ad83-2e5327222014" containerName="main" Jul 28 17:43:31.516661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.516512 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe34ad27-a690-4777-ae5f-fa48b014c658" containerName="main" Jul 28 17:43:31.521301 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.521278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.526212 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.526184 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\"" Jul 28 17:43:31.526212 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.526199 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\"" Jul 28 17:43:31.526423 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.526187 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\"" Jul 28 17:43:31.526423 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.526187 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-fnftk\"" Jul 28 17:43:31.538214 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.538181 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx"] Jul 28 17:43:31.629110 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.629069 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4bx\" (UniqueName: \"kubernetes.io/projected/4b0d5b1a-eaf1-4639-988d-76514090c584-kube-api-access-qk4bx\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.629110 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.629112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.629327 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.629226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0d5b1a-eaf1-4639-988d-76514090c584-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.629327 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.629255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.629327 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.629272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.629327 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.629299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730435 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0d5b1a-eaf1-4639-988d-76514090c584-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730435 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4bx\" (UniqueName: \"kubernetes.io/projected/4b0d5b1a-eaf1-4639-988d-76514090c584-kube-api-access-qk4bx\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.730950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.731007 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.731007 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.730989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.732880 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.732858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.733180 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.733161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0d5b1a-eaf1-4639-988d-76514090c584-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.744680 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.744649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4bx\" (UniqueName: \"kubernetes.io/projected/4b0d5b1a-eaf1-4639-988d-76514090c584-kube-api-access-qk4bx\") pod \"scheduler-inline-config-test-kserve-8655bc757-bzccx\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:31.831824 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:31.831790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:32.002205 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:32.002175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx"] Jul 28 17:43:32.002496 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:43:32.002473 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0d5b1a_eaf1_4639_988d_76514090c584.slice/crio-167b389901bb55343836a1f3ad468eb7f231d48f2b18fdaa04f2f26e3e210896 WatchSource:0}: Error finding container 167b389901bb55343836a1f3ad468eb7f231d48f2b18fdaa04f2f26e3e210896: Status 404 returned error can't find the container with id 167b389901bb55343836a1f3ad468eb7f231d48f2b18fdaa04f2f26e3e210896 Jul 28 17:43:32.557196 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:32.557155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" event={"ID":"4b0d5b1a-eaf1-4639-988d-76514090c584","Type":"ContainerStarted","Data":"0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba"} Jul 28 17:43:32.557682 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:32.557222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" event={"ID":"4b0d5b1a-eaf1-4639-988d-76514090c584","Type":"ContainerStarted","Data":"167b389901bb55343836a1f3ad468eb7f231d48f2b18fdaa04f2f26e3e210896"} Jul 28 17:43:32.581569 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:32.581512 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" podStartSLOduration=1.58149297 podStartE2EDuration="1.58149297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:32.580133548 +0000 UTC m=+1245.732633725" watchObservedRunningTime="2026-07-28 17:43:32.58149297 +0000 UTC m=+1245.733993203" Jul 28 17:43:32.832205 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:32.832175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:41.831992 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:41.831953 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:41.844491 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:41.844457 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:43:42.621891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:43:42.621861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:44:05.271518 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:05.271481 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx"] Jul 28 17:44:05.281563 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:44:05.281524 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:44:05.281563 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:44:05.281563 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:44:05.281563 ip-10-0-131-29 kubenswrapper[2564]: > containerID="0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba" cmd=["/bin/sleep","15"] Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:05.281581 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" message="" err=< Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:44:05.281593 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerName="main" containerID="cri-o://0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba" Jul 28 17:44:05.281862 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:05.281641 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerName="main" containerID="cri-o://0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba" gracePeriod=60 Jul 28 17:44:05.611477 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:05.611437 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Jul 28 17:44:05.722593 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:05.722562 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerID="0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba" exitCode=0 Jul 28 17:44:05.722793 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:05.722640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" event={"ID":"4b0d5b1a-eaf1-4639-988d-76514090c584","Type":"ContainerDied","Data":"0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba"} Jul 28 17:44:06.033363 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.033333 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:44:06.149623 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149528 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0d5b1a-eaf1-4639-988d-76514090c584-tls-certs\") pod \"4b0d5b1a-eaf1-4639-988d-76514090c584\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " Jul 28 17:44:06.149623 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149579 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-tmp-dir\") pod \"4b0d5b1a-eaf1-4639-988d-76514090c584\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " Jul 28 17:44:06.150093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149646 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-model-cache\") pod \"4b0d5b1a-eaf1-4639-988d-76514090c584\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " Jul 28 17:44:06.150093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149682 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk4bx\" (UniqueName: \"kubernetes.io/projected/4b0d5b1a-eaf1-4639-988d-76514090c584-kube-api-access-qk4bx\") pod \"4b0d5b1a-eaf1-4639-988d-76514090c584\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " Jul 28 17:44:06.150093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-dshm\") pod \"4b0d5b1a-eaf1-4639-988d-76514090c584\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " Jul 28 17:44:06.150093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-home\") pod \"4b0d5b1a-eaf1-4639-988d-76514090c584\" (UID: \"4b0d5b1a-eaf1-4639-988d-76514090c584\") " Jul 28 17:44:06.150093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149939 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b0d5b1a-eaf1-4639-988d-76514090c584" (UID: "4b0d5b1a-eaf1-4639-988d-76514090c584"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:44:06.150093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.149936 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b0d5b1a-eaf1-4639-988d-76514090c584" (UID: "4b0d5b1a-eaf1-4639-988d-76514090c584"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:44:06.150440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.150119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-home" (OuterVolumeSpecName: "home") pod "4b0d5b1a-eaf1-4639-988d-76514090c584" (UID: "4b0d5b1a-eaf1-4639-988d-76514090c584"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:44:06.150440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.150223 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:44:06.150440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.150249 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:44:06.150440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.150262 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:44:06.152346 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.152311 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0d5b1a-eaf1-4639-988d-76514090c584-kube-api-access-qk4bx" (OuterVolumeSpecName: "kube-api-access-qk4bx") pod "4b0d5b1a-eaf1-4639-988d-76514090c584" (UID: "4b0d5b1a-eaf1-4639-988d-76514090c584"). InnerVolumeSpecName "kube-api-access-qk4bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:44:06.152346 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.152310 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-dshm" (OuterVolumeSpecName: "dshm") pod "4b0d5b1a-eaf1-4639-988d-76514090c584" (UID: "4b0d5b1a-eaf1-4639-988d-76514090c584"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:44:06.152509 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.152377 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0d5b1a-eaf1-4639-988d-76514090c584-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b0d5b1a-eaf1-4639-988d-76514090c584" (UID: "4b0d5b1a-eaf1-4639-988d-76514090c584"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:44:06.251056 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.251016 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0d5b1a-eaf1-4639-988d-76514090c584-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:44:06.251056 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.251040 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk4bx\" (UniqueName: \"kubernetes.io/projected/4b0d5b1a-eaf1-4639-988d-76514090c584-kube-api-access-qk4bx\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:44:06.251056 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.251049 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0d5b1a-eaf1-4639-988d-76514090c584-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:44:06.729028 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.728999 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" Jul 28 17:44:06.729450 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.729040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx" event={"ID":"4b0d5b1a-eaf1-4639-988d-76514090c584","Type":"ContainerDied","Data":"167b389901bb55343836a1f3ad468eb7f231d48f2b18fdaa04f2f26e3e210896"} Jul 28 17:44:06.729450 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.729098 2564 scope.go:117] "RemoveContainer" containerID="0f1a6d353f1d57b4bf10d40f676a57060c6db56341c5ad9ca1a4c343aeb8a6ba" Jul 28 17:44:06.751488 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.751457 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx"] Jul 28 17:44:06.755263 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:06.755238 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-bzccx"] Jul 28 17:44:07.384103 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:07.384062 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" path="/var/lib/kubelet/pods/4b0d5b1a-eaf1-4639-988d-76514090c584/volumes" Jul 28 17:44:48.391206 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.391172 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h"] Jul 28 17:44:48.391952 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.391590 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerName="main" Jul 28 17:44:48.391952 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.391600 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerName="main" Jul 28 17:44:48.391952 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.391665 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0d5b1a-eaf1-4639-988d-76514090c584" containerName="main" Jul 28 17:44:48.395170 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.395149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.399644 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.399622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-ckfkd\"" Jul 28 17:44:48.399787 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.399622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\"" Jul 28 17:44:48.399787 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.399622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\"" Jul 28 17:44:48.399787 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.399706 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\"" Jul 28 17:44:48.406785 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.406742 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h"] Jul 28 17:44:48.537474 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.537423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.537474 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.537481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985lz\" (UniqueName: \"kubernetes.io/projected/8c2a29d5-8858-4a7f-b92c-aea122a5807d-kube-api-access-985lz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.537709 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.537532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.537709 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.537567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.537709 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.537606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.537868 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.537706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638493 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985lz\" (UniqueName: \"kubernetes.io/projected/8c2a29d5-8858-4a7f-b92c-aea122a5807d-kube-api-access-985lz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638686 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.638985 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.638960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.639051 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.639003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.639186 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.639166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.640884 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.640863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.641354 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.641281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.651476 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.651450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-985lz\" (UniqueName: \"kubernetes.io/projected/8c2a29d5-8858-4a7f-b92c-aea122a5807d-kube-api-access-985lz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.706328 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.706284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:48.851290 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.851226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h"] Jul 28 17:44:48.854120 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:44:48.854093 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2a29d5_8858_4a7f_b92c_aea122a5807d.slice/crio-5ef2463e681a5653b4796c5a6781cece3080c6f061cd195f36c6a7b01817a847 WatchSource:0}: Error finding container 5ef2463e681a5653b4796c5a6781cece3080c6f061cd195f36c6a7b01817a847: Status 404 returned error can't find the container with id 5ef2463e681a5653b4796c5a6781cece3080c6f061cd195f36c6a7b01817a847 Jul 28 17:44:48.937904 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.937870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" event={"ID":"8c2a29d5-8858-4a7f-b92c-aea122a5807d","Type":"ContainerStarted","Data":"055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9"} Jul 28 17:44:48.938108 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.937922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" event={"ID":"8c2a29d5-8858-4a7f-b92c-aea122a5807d","Type":"ContainerStarted","Data":"5ef2463e681a5653b4796c5a6781cece3080c6f061cd195f36c6a7b01817a847"} Jul 28 17:44:48.958720 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:48.958644 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" podStartSLOduration=0.958627314 podStartE2EDuration="958.627314ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:48.956774387 +0000 UTC m=+1322.109274584" watchObservedRunningTime="2026-07-28 17:44:48.958627314 +0000 UTC m=+1322.111127491" Jul 28 17:44:49.707072 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:49.707030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:58.707251 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:58.707196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:58.719802 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:58.719748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:44:59.002312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:44:59.002221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:45:21.537788 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.537686 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h"] Jul 28 17:45:21.547780 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:45:21.547718 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:45:21.547780 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:45:21.547780 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:45:21.547780 ip-10-0-131-29 kubenswrapper[2564]: > containerID="055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" cmd=["/bin/sleep","15"] Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.547795 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" message="" err=< Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:45:21.547812 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" podUID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" containerName="main" containerID="cri-o://055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" Jul 28 17:45:21.548034 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.547871 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" podUID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" containerName="main" containerID="cri-o://055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" gracePeriod=60 Jul 28 17:45:21.905297 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.905270 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:45:21.994502 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-home\") pod \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " Jul 28 17:45:21.994502 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-985lz\" (UniqueName: \"kubernetes.io/projected/8c2a29d5-8858-4a7f-b92c-aea122a5807d-kube-api-access-985lz\") pod \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " Jul 28 17:45:21.994799 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tls-certs\") pod \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " Jul 28 17:45:21.994799 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-model-cache\") pod \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " Jul 28 17:45:21.994799 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tmp-dir\") pod \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " Jul 28 17:45:21.994799 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-dshm\") pod \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\" (UID: \"8c2a29d5-8858-4a7f-b92c-aea122a5807d\") " Jul 28 17:45:21.995004 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994869 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-home" (OuterVolumeSpecName: "home") pod "8c2a29d5-8858-4a7f-b92c-aea122a5807d" (UID: "8c2a29d5-8858-4a7f-b92c-aea122a5807d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:21.995004 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.994938 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c2a29d5-8858-4a7f-b92c-aea122a5807d" (UID: "8c2a29d5-8858-4a7f-b92c-aea122a5807d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:21.995104 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.995018 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c2a29d5-8858-4a7f-b92c-aea122a5807d" (UID: "8c2a29d5-8858-4a7f-b92c-aea122a5807d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:21.996961 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.996935 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c2a29d5-8858-4a7f-b92c-aea122a5807d" (UID: "8c2a29d5-8858-4a7f-b92c-aea122a5807d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:45:21.997061 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.996950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-dshm" (OuterVolumeSpecName: "dshm") pod "8c2a29d5-8858-4a7f-b92c-aea122a5807d" (UID: "8c2a29d5-8858-4a7f-b92c-aea122a5807d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:21.997061 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:21.996983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a29d5-8858-4a7f-b92c-aea122a5807d-kube-api-access-985lz" (OuterVolumeSpecName: "kube-api-access-985lz") pod "8c2a29d5-8858-4a7f-b92c-aea122a5807d" (UID: "8c2a29d5-8858-4a7f-b92c-aea122a5807d"). InnerVolumeSpecName "kube-api-access-985lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:45:22.095615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.095569 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:45:22.095615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.095603 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:45:22.095615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.095612 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:45:22.095615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.095620 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:45:22.095615 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.095628 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a29d5-8858-4a7f-b92c-aea122a5807d-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:45:22.095961 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.095637 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-985lz\" (UniqueName: \"kubernetes.io/projected/8c2a29d5-8858-4a7f-b92c-aea122a5807d-kube-api-access-985lz\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:45:22.107999 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.107969 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" containerID="055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" exitCode=0 Jul 28 17:45:22.108147 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.108016 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" Jul 28 17:45:22.108147 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.108049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" event={"ID":"8c2a29d5-8858-4a7f-b92c-aea122a5807d","Type":"ContainerDied","Data":"055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9"} Jul 28 17:45:22.108147 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.108103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h" event={"ID":"8c2a29d5-8858-4a7f-b92c-aea122a5807d","Type":"ContainerDied","Data":"5ef2463e681a5653b4796c5a6781cece3080c6f061cd195f36c6a7b01817a847"} Jul 28 17:45:22.108147 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.108126 2564 scope.go:117] "RemoveContainer" containerID="055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" Jul 28 17:45:22.117885 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.117858 2564 scope.go:117] "RemoveContainer" containerID="055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" Jul 28 17:45:22.118214 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:45:22.118191 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9\": container with ID starting with 055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9 not found: ID does not exist" containerID="055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9" Jul 28 17:45:22.118312 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.118222 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9"} err="failed to get container status \"055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9\": rpc error: code = NotFound desc = could not find container \"055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9\": container with ID starting with 055d7c5e8bb5732459ade45b5526e02a816c2655a4068acdc9b76e5eef4d0bc9 not found: ID does not exist" Jul 28 17:45:22.130732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.130703 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h"] Jul 28 17:45:22.139318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:22.139282 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4ph8h"] Jul 28 17:45:23.376886 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:23.376852 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" path="/var/lib/kubelet/pods/8c2a29d5-8858-4a7f-b92c-aea122a5807d/volumes" Jul 28 17:45:35.956854 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.956808 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs"] Jul 28 17:45:35.957671 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.957649 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" containerName="main" Jul 28 17:45:35.957792 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.957675 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" containerName="main" Jul 28 17:45:35.957887 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.957873 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2a29d5-8858-4a7f-b92c-aea122a5807d" containerName="main" Jul 28 17:45:35.963747 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.963721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:35.966672 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.966641 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\"" Jul 28 17:45:35.966834 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.966726 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\"" Jul 28 17:45:35.966834 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.966753 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\"" Jul 28 17:45:35.967731 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.967717 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\"" Jul 28 17:45:35.967795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.967732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-nd8p6\"" Jul 28 17:45:35.974191 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:35.974166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs"] Jul 28 17:45:36.019816 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.019779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-home\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.019997 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.019864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-dshm\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.019997 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.019911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tls-certs\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.019997 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.019940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-model-cache\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.019997 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.019956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kserve-provision-location\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.020139 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.020044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tmp-dir\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.020139 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.020074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhw5q\" (UniqueName: \"kubernetes.io/projected/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kube-api-access-fhw5q\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120670 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tls-certs\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120670 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-model-cache\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kserve-provision-location\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tmp-dir\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhw5q\" (UniqueName: \"kubernetes.io/projected/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kube-api-access-fhw5q\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-home\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.120950 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.120895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-dshm\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.121209 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.121127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-model-cache\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.121209 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.121184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kserve-provision-location\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.121304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.121233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tmp-dir\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.121304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.121273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-home\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.123050 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.123028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-dshm\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.123214 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.123197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tls-certs\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.130204 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.130179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhw5q\" (UniqueName: \"kubernetes.io/projected/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kube-api-access-fhw5q\") pod \"auth-disabled-test-kserve-84794c675c-d66rs\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.274960 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.274877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:36.429610 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:36.429577 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs"] Jul 28 17:45:36.432010 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:45:36.431983 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea19fbb_66b9_4416_9a44_4b76d2ac6329.slice/crio-75a136ba93844fc93fd2df5c7b798b88be7cd13dd485a783bf6792e975977cbe WatchSource:0}: Error finding container 75a136ba93844fc93fd2df5c7b798b88be7cd13dd485a783bf6792e975977cbe: Status 404 returned error can't find the container with id 75a136ba93844fc93fd2df5c7b798b88be7cd13dd485a783bf6792e975977cbe Jul 28 17:45:37.183482 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:37.183448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" event={"ID":"8ea19fbb-66b9-4416-9a44-4b76d2ac6329","Type":"ContainerStarted","Data":"70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4"} Jul 28 17:45:37.183907 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:37.183507 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" event={"ID":"8ea19fbb-66b9-4416-9a44-4b76d2ac6329","Type":"ContainerStarted","Data":"75a136ba93844fc93fd2df5c7b798b88be7cd13dd485a783bf6792e975977cbe"} Jul 28 17:45:42.209234 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:42.209195 2564 generic.go:356] "Generic (PLEG): container finished" podID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerID="70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4" exitCode=0 Jul 28 17:45:42.209724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:42.209269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" event={"ID":"8ea19fbb-66b9-4416-9a44-4b76d2ac6329","Type":"ContainerDied","Data":"70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4"} Jul 28 17:45:43.216372 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:43.216338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" event={"ID":"8ea19fbb-66b9-4416-9a44-4b76d2ac6329","Type":"ContainerStarted","Data":"bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5"} Jul 28 17:45:43.251648 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:43.251586 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podStartSLOduration=3.076691522 podStartE2EDuration="8.251569795s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.174878273s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:43.249969472 +0000 UTC m=+1376.402469684" watchObservedRunningTime="2026-07-28 17:45:43.251569795 +0000 UTC m=+1376.404069971" Jul 28 17:45:43.275850 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:43.275817 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:46.275693 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:46.275651 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:45:46.277149 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:46.277111 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:45:56.275443 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:45:56.275387 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:46:06.275440 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:46:06.275387 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:46:16.275931 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:46:16.275887 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:46:26.276304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:46:26.276250 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:46:36.275647 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:46:36.275601 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:46:46.276304 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:46:46.276244 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:46:56.276030 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:46:56.275979 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:47:06.276153 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:06.276094 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:47:16.275382 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:16.275339 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:47:26.276167 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:26.276130 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:47:36.276285 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:36.276230 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused" Jul 28 17:47:46.286333 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:46.286299 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:47:46.294448 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:46.294419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:47:47.462380 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:47.462348 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:47:47.467095 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:47.467062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:47:51.531412 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:47:51.531375 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs"] Jul 28 17:48:06.553609 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:06.553542 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" containerID="cri-o://bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5" gracePeriod=45 Jul 28 17:48:07.313177 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.313144 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:48:07.452201 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452168 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kserve-provision-location\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452415 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-model-cache\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452415 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452251 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tmp-dir\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452415 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452285 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-dshm\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452415 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452324 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-home\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452415 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452341 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhw5q\" (UniqueName: \"kubernetes.io/projected/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kube-api-access-fhw5q\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452415 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452406 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tls-certs\") pod \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\" (UID: \"8ea19fbb-66b9-4416-9a44-4b76d2ac6329\") " Jul 28 17:48:07.452726 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452589 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:48:07.452726 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452704 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.453013 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.452988 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-home" (OuterVolumeSpecName: "home") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:48:07.454674 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.454647 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-dshm" (OuterVolumeSpecName: "dshm") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:48:07.454964 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.454930 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kube-api-access-fhw5q" (OuterVolumeSpecName: "kube-api-access-fhw5q") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "kube-api-access-fhw5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:48:07.455186 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.455169 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:48:07.466331 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.466294 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:48:07.514206 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.514160 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ea19fbb-66b9-4416-9a44-4b76d2ac6329" (UID: "8ea19fbb-66b9-4416-9a44-4b76d2ac6329"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:48:07.553865 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.553829 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.553865 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.553861 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.554267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.553873 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.554267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.553883 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.554267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.553894 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.554267 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.553902 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhw5q\" (UniqueName: \"kubernetes.io/projected/8ea19fbb-66b9-4416-9a44-4b76d2ac6329-kube-api-access-fhw5q\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:48:07.986197 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.986160 2564 generic.go:356] "Generic (PLEG): container finished" podID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerID="bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5" exitCode=0 Jul 28 17:48:07.986374 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.986208 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" Jul 28 17:48:07.986374 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.986242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" event={"ID":"8ea19fbb-66b9-4416-9a44-4b76d2ac6329","Type":"ContainerDied","Data":"bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5"} Jul 28 17:48:07.986374 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.986299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" event={"ID":"8ea19fbb-66b9-4416-9a44-4b76d2ac6329","Type":"ContainerDied","Data":"75a136ba93844fc93fd2df5c7b798b88be7cd13dd485a783bf6792e975977cbe"} Jul 28 17:48:07.986374 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.986332 2564 scope.go:117] "RemoveContainer" containerID="bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5" Jul 28 17:48:07.995556 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:07.995541 2564 scope.go:117] "RemoveContainer" containerID="70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4" Jul 28 17:48:08.003810 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.003789 2564 scope.go:117] "RemoveContainer" containerID="bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5" Jul 28 17:48:08.004079 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:48:08.004061 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5\": container with ID starting with bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5 not found: ID does not exist" containerID="bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5" Jul 28 17:48:08.004131 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.004089 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5"} err="failed to get container status \"bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5\": rpc error: code = NotFound desc = could not find container \"bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5\": container with ID starting with bd78f00bcfa35505ebd59999dddbfc2700fe3dfd7f7eb238f339f4243e20c3f5 not found: ID does not exist" Jul 28 17:48:08.004131 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.004108 2564 scope.go:117] "RemoveContainer" containerID="70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4" Jul 28 17:48:08.004371 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:48:08.004354 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4\": container with ID starting with 70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4 not found: ID does not exist" containerID="70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4" Jul 28 17:48:08.004414 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.004378 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4"} err="failed to get container status \"70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4\": rpc error: code = NotFound desc = could not find container \"70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4\": container with ID starting with 70184a674eb430e52777611d0916d38d462d4516dc9eef7c8f7d77b95c74f8f4 not found: ID does not exist" Jul 28 17:48:08.010417 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.010393 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs"] Jul 28 17:48:08.014508 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.014486 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs"] Jul 28 17:48:08.287016 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:08.286907 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-d66rs" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Jul 28 17:48:09.376486 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:09.376410 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" path="/var/lib/kubelet/pods/8ea19fbb-66b9-4416-9a44-4b76d2ac6329/volumes" Jul 28 17:48:20.621946 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.621909 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb"] Jul 28 17:48:20.622391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.622304 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="storage-initializer" Jul 28 17:48:20.622391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.622315 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="storage-initializer" Jul 28 17:48:20.622391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.622331 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" Jul 28 17:48:20.622391 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.622337 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" Jul 28 17:48:20.622520 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.622401 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea19fbb-66b9-4416-9a44-4b76d2ac6329" containerName="main" Jul 28 17:48:20.625279 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.625261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.628742 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.628714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\"" Jul 28 17:48:20.629227 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.629211 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\"" Jul 28 17:48:20.631037 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.630556 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\"" Jul 28 17:48:20.631037 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.630564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-8cfh9\"" Jul 28 17:48:20.647804 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.647771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb"] Jul 28 17:48:20.771866 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.771831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6b41c-19b6-4fce-a819-81229b680ca6-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.771866 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.771867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.772080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.771887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njp9\" (UniqueName: \"kubernetes.io/projected/9bf6b41c-19b6-4fce-a819-81229b680ca6-kube-api-access-7njp9\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.772080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.771966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.772080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.772021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.772080 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.772075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873187 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873187 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873398 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6b41c-19b6-4fce-a819-81229b680ca6-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873398 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njp9\" (UniqueName: \"kubernetes.io/projected/9bf6b41c-19b6-4fce-a819-81229b680ca6-kube-api-access-7njp9\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873504 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873608 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873661 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873624 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.873795 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.873774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.875461 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.875432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.875620 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.875602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6b41c-19b6-4fce-a819-81229b680ca6-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.885952 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.885929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njp9\" (UniqueName: \"kubernetes.io/projected/9bf6b41c-19b6-4fce-a819-81229b680ca6-kube-api-access-7njp9\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:20.935406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:20.935377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:21.084031 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:21.084004 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb"] Jul 28 17:48:21.084724 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:48:21.084692 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf6b41c_19b6_4fce_a819_81229b680ca6.slice/crio-ac61d22acf599799b3ec01c17134190252bd43eeddfdcbd25f463f8afaa4b457 WatchSource:0}: Error finding container ac61d22acf599799b3ec01c17134190252bd43eeddfdcbd25f463f8afaa4b457: Status 404 returned error can't find the container with id ac61d22acf599799b3ec01c17134190252bd43eeddfdcbd25f463f8afaa4b457 Jul 28 17:48:22.054610 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:22.054574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" event={"ID":"9bf6b41c-19b6-4fce-a819-81229b680ca6","Type":"ContainerStarted","Data":"edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622"} Jul 28 17:48:22.055015 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:22.054628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" event={"ID":"9bf6b41c-19b6-4fce-a819-81229b680ca6","Type":"ContainerStarted","Data":"ac61d22acf599799b3ec01c17134190252bd43eeddfdcbd25f463f8afaa4b457"} Jul 28 17:48:22.935957 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:22.935914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:30.936129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:30.936088 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:30.949134 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:30.949103 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:48:30.973336 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:30.973278 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" podStartSLOduration=10.973260796 podStartE2EDuration="10.973260796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:22.137155203 +0000 UTC m=+1535.289655409" watchObservedRunningTime="2026-07-28 17:48:30.973260796 +0000 UTC m=+1544.125760973" Jul 28 17:48:31.111045 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:48:31.111011 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 17:52:17.765483 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.765440 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn"] Jul 28 17:52:17.769112 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.769089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.778097 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.778064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\"" Jul 28 17:52:17.778460 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.778434 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\"" Jul 28 17:52:17.779137 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.779114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-fx57c\"" Jul 28 17:52:17.779257 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.779114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\"" Jul 28 17:52:17.779257 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.779226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\"" Jul 28 17:52:17.807002 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.806966 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn"] Jul 28 17:52:17.913339 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.913339 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.913564 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdq7\" (UniqueName: \"kubernetes.io/projected/7f464543-7ca9-437e-9480-75e10460781e-kube-api-access-gpdq7\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.913564 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.913564 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.913564 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:17.913701 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:17.913571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f464543-7ca9-437e-9480-75e10460781e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.014665 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.014632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f464543-7ca9-437e-9480-75e10460781e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.014887 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.014685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.014887 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.014707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.014887 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.014723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdq7\" (UniqueName: \"kubernetes.io/projected/7f464543-7ca9-437e-9480-75e10460781e-kube-api-access-gpdq7\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.014887 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.014742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.015082 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.014915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.015082 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.015040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.015326 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.015295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.015326 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.015314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.015549 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.015375 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.015549 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.015475 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.017065 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.017042 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.017464 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.017444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f464543-7ca9-437e-9480-75e10460781e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.026735 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.026702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdq7\" (UniqueName: \"kubernetes.io/projected/7f464543-7ca9-437e-9480-75e10460781e-kube-api-access-gpdq7\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-657wn\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.079984 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.079940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:18.278132 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:18.278103 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn"] Jul 28 17:52:18.280150 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:52:18.280121 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f464543_7ca9_437e_9480_75e10460781e.slice/crio-ae726ea3872f952c402e8a25cd1aa1652187697dfeb70d15ad3b3452812f171e WatchSource:0}: Error finding container ae726ea3872f952c402e8a25cd1aa1652187697dfeb70d15ad3b3452812f171e: Status 404 returned error can't find the container with id ae726ea3872f952c402e8a25cd1aa1652187697dfeb70d15ad3b3452812f171e Jul 28 17:52:19.269077 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:19.269025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" event={"ID":"7f464543-7ca9-437e-9480-75e10460781e","Type":"ContainerStarted","Data":"b209cf1702d42b6a40e50da1ee1a5233b41ff0cb055ef68e62ae61a82121eb8b"} Jul 28 17:52:19.269594 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:19.269093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" event={"ID":"7f464543-7ca9-437e-9480-75e10460781e","Type":"ContainerStarted","Data":"ae726ea3872f952c402e8a25cd1aa1652187697dfeb70d15ad3b3452812f171e"} Jul 28 17:52:24.298093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:24.298059 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f464543-7ca9-437e-9480-75e10460781e" containerID="b209cf1702d42b6a40e50da1ee1a5233b41ff0cb055ef68e62ae61a82121eb8b" exitCode=0 Jul 28 17:52:24.298093 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:24.298095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" event={"ID":"7f464543-7ca9-437e-9480-75e10460781e","Type":"ContainerDied","Data":"b209cf1702d42b6a40e50da1ee1a5233b41ff0cb055ef68e62ae61a82121eb8b"} Jul 28 17:52:25.306346 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:25.306307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" event={"ID":"7f464543-7ca9-437e-9480-75e10460781e","Type":"ContainerStarted","Data":"650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b"} Jul 28 17:52:25.346925 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:25.346871 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" podStartSLOduration=3.098543494 podStartE2EDuration="8.346852595s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.248309101s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:25.344043265 +0000 UTC m=+1778.496543443" watchObservedRunningTime="2026-07-28 17:52:25.346852595 +0000 UTC m=+1778.499352789" Jul 28 17:52:26.081082 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:26.081042 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:28.080338 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:28.080299 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:28.093289 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:28.093253 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:28.332751 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:28.332661 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:52:47.506422 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:47.506391 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:52:47.508863 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:52:47.508843 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:53:32.289628 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.289594 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn"] Jul 28 17:53:32.299000 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:53:32.298954 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:53:32.299000 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:53:32.299000 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:53:32.299000 ip-10-0-131-29 kubenswrapper[2564]: > containerID="650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b" cmd=["/bin/sleep","15"] Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.299020 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" message="" err=< Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:53:32.299033 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="main" containerID="cri-o://650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b" Jul 28 17:53:32.299348 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.299074 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="main" containerID="cri-o://650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b" gracePeriod=60 Jul 28 17:53:32.322358 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.322317 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Jul 28 17:53:32.642827 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.642740 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f464543-7ca9-437e-9480-75e10460781e" containerID="650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b" exitCode=0 Jul 28 17:53:32.642827 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.642794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" event={"ID":"7f464543-7ca9-437e-9480-75e10460781e","Type":"ContainerDied","Data":"650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b"} Jul 28 17:53:32.643032 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.642891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" event={"ID":"7f464543-7ca9-437e-9480-75e10460781e","Type":"ContainerDied","Data":"ae726ea3872f952c402e8a25cd1aa1652187697dfeb70d15ad3b3452812f171e"} Jul 28 17:53:32.643032 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.642914 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae726ea3872f952c402e8a25cd1aa1652187697dfeb70d15ad3b3452812f171e" Jul 28 17:53:32.658703 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.658676 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:53:32.720966 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.720935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f464543-7ca9-437e-9480-75e10460781e-tls-certs\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.720978 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-dshm\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.720997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-model-cache\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-kserve-provision-location\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721104 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpdq7\" (UniqueName: \"kubernetes.io/projected/7f464543-7ca9-437e-9480-75e10460781e-kube-api-access-gpdq7\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721148 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721131 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-tmp-dir\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721412 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721188 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-home\") pod \"7f464543-7ca9-437e-9480-75e10460781e\" (UID: \"7f464543-7ca9-437e-9480-75e10460781e\") " Jul 28 17:53:32.721412 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721341 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:32.721530 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721479 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:32.721589 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721572 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-home" (OuterVolumeSpecName: "home") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:32.721753 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.721722 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:32.723533 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.723479 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f464543-7ca9-437e-9480-75e10460781e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:53:32.723634 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.723540 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-dshm" (OuterVolumeSpecName: "dshm") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:32.723634 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.723575 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f464543-7ca9-437e-9480-75e10460781e-kube-api-access-gpdq7" (OuterVolumeSpecName: "kube-api-access-gpdq7") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "kube-api-access-gpdq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:53:32.789443 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.789386 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f464543-7ca9-437e-9480-75e10460781e" (UID: "7f464543-7ca9-437e-9480-75e10460781e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:32.822453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.822363 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:32.822453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.822394 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f464543-7ca9-437e-9480-75e10460781e-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:32.822453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.822408 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:32.822453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.822421 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:32.822453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.822431 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdq7\" (UniqueName: \"kubernetes.io/projected/7f464543-7ca9-437e-9480-75e10460781e-kube-api-access-gpdq7\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:32.822453 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:32.822440 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f464543-7ca9-437e-9480-75e10460781e-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:53:33.647807 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:33.647703 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn" Jul 28 17:53:33.669514 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:33.669483 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn"] Jul 28 17:53:33.673581 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:33.673554 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-657wn"] Jul 28 17:53:35.377536 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:35.377501 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f464543-7ca9-437e-9480-75e10460781e" path="/var/lib/kubelet/pods/7f464543-7ca9-437e-9480-75e10460781e/volumes" Jul 28 17:53:57.279458 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.279419 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn"] Jul 28 17:53:57.279977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.279895 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="main" Jul 28 17:53:57.279977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.279910 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="main" Jul 28 17:53:57.279977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.279938 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="storage-initializer" Jul 28 17:53:57.279977 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.279944 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="storage-initializer" Jul 28 17:53:57.280129 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.280006 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f464543-7ca9-437e-9480-75e10460781e" containerName="main" Jul 28 17:53:57.283278 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.283258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.286936 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.286914 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\"" Jul 28 17:53:57.288248 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.288232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-lpvmq\"" Jul 28 17:53:57.288248 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.288240 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\"" Jul 28 17:53:57.288400 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.288244 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\"" Jul 28 17:53:57.288400 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.288296 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\"" Jul 28 17:53:57.297598 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.297571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn"] Jul 28 17:53:57.350127 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.350127 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.350341 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.350341 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.350341 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.350341 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.350477 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.350385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np246\" (UniqueName: \"kubernetes.io/projected/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kube-api-access-np246\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451124 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np246\" (UniqueName: \"kubernetes.io/projected/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kube-api-access-np246\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451318 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451664 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451636 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451720 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451790 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.451837 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.451805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.453593 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.453573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.453865 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.453849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.463350 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.463324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np246\" (UniqueName: \"kubernetes.io/projected/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kube-api-access-np246\") pod \"tokenizer-migration-path-test-kserve-6666787596-cj2jn\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.594157 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.594123 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:53:57.750576 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.750551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn"] Jul 28 17:53:57.753273 ip-10-0-131-29 kubenswrapper[2564]: W0728 17:53:57.753245 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd34e1a5_b0a3_4612_84bf_f211db085a4c.slice/crio-d87d03adc057aa106d67babae808fccb7c242bbfb60bb23855c783ef5469f54b WatchSource:0}: Error finding container d87d03adc057aa106d67babae808fccb7c242bbfb60bb23855c783ef5469f54b: Status 404 returned error can't find the container with id d87d03adc057aa106d67babae808fccb7c242bbfb60bb23855c783ef5469f54b Jul 28 17:53:57.776386 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:57.776359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" event={"ID":"dd34e1a5-b0a3-4612-84bf-f211db085a4c","Type":"ContainerStarted","Data":"d87d03adc057aa106d67babae808fccb7c242bbfb60bb23855c783ef5469f54b"} Jul 28 17:53:58.785406 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:53:58.785358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" event={"ID":"dd34e1a5-b0a3-4612-84bf-f211db085a4c","Type":"ContainerStarted","Data":"deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0"} Jul 28 17:54:02.808439 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:02.808332 2564 generic.go:356] "Generic (PLEG): container finished" podID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerID="deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0" exitCode=0 Jul 28 17:54:02.808439 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:02.808414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" event={"ID":"dd34e1a5-b0a3-4612-84bf-f211db085a4c","Type":"ContainerDied","Data":"deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0"} Jul 28 17:54:03.819193 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:03.819156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" event={"ID":"dd34e1a5-b0a3-4612-84bf-f211db085a4c","Type":"ContainerStarted","Data":"ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca"} Jul 28 17:54:03.840475 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:03.840416 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" podStartSLOduration=2.273423427 podStartE2EDuration="6.840398115s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.566974688s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:03.839709447 +0000 UTC m=+1876.992209627" watchObservedRunningTime="2026-07-28 17:54:03.840398115 +0000 UTC m=+1876.992898291" Jul 28 17:54:04.595003 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:04.594956 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:54:07.595354 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:07.595311 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:54:07.607904 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:07.607873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:54:07.852264 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:07.852172 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:54:37.237017 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.236979 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn"] Jul 28 17:54:37.249131 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:54:37.249084 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 17:54:37.249131 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:54:37.249131 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:54:37.249131 ip-10-0-131-29 kubenswrapper[2564]: > containerID="ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" cmd=["/bin/sleep","15"] Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.249149 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" message="" err=< Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:54:37.249162 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="main" containerID="cri-o://ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" Jul 28 17:54:37.249463 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.249209 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="main" containerID="cri-o://ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" gracePeriod=60 Jul 28 17:54:37.612885 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.612856 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:54:37.714289 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tls-certs\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714510 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-home\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714510 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714373 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np246\" (UniqueName: \"kubernetes.io/projected/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kube-api-access-np246\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714510 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714436 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kserve-provision-location\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714510 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714454 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tmp-dir\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-model-cache\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714597 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-home" (OuterVolumeSpecName: "home") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:37.714732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-dshm\") pod \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\" (UID: \"dd34e1a5-b0a3-4612-84bf-f211db085a4c\") " Jul 28 17:54:37.714732 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.714685 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:37.715107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.715049 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.715107 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.715073 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.715307 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.715178 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:37.717236 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.717200 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kube-api-access-np246" (OuterVolumeSpecName: "kube-api-access-np246") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "kube-api-access-np246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:54:37.717483 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.717461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:54:37.717557 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.717469 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-dshm" (OuterVolumeSpecName: "dshm") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:37.785337 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.785234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd34e1a5-b0a3-4612-84bf-f211db085a4c" (UID: "dd34e1a5-b0a3-4612-84bf-f211db085a4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:37.816004 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.815964 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.816004 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.816000 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.816004 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.816008 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd34e1a5-b0a3-4612-84bf-f211db085a4c-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.816252 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.816017 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np246\" (UniqueName: \"kubernetes.io/projected/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kube-api-access-np246\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.816252 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.816029 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd34e1a5-b0a3-4612-84bf-f211db085a4c-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 17:54:37.996698 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.996664 2564 generic.go:356] "Generic (PLEG): container finished" podID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerID="ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" exitCode=0 Jul 28 17:54:37.996891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.996721 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" Jul 28 17:54:37.996891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.996777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" event={"ID":"dd34e1a5-b0a3-4612-84bf-f211db085a4c","Type":"ContainerDied","Data":"ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca"} Jul 28 17:54:37.996891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.996836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn" event={"ID":"dd34e1a5-b0a3-4612-84bf-f211db085a4c","Type":"ContainerDied","Data":"d87d03adc057aa106d67babae808fccb7c242bbfb60bb23855c783ef5469f54b"} Jul 28 17:54:37.996891 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:37.996864 2564 scope.go:117] "RemoveContainer" containerID="ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" Jul 28 17:54:38.006981 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.006962 2564 scope.go:117] "RemoveContainer" containerID="deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0" Jul 28 17:54:38.016103 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.016082 2564 scope.go:117] "RemoveContainer" containerID="ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" Jul 28 17:54:38.016432 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:54:38.016412 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca\": container with ID starting with ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca not found: ID does not exist" containerID="ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca" Jul 28 17:54:38.016515 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.016445 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca"} err="failed to get container status \"ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca\": rpc error: code = NotFound desc = could not find container \"ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca\": container with ID starting with ffa105cb0191c43ad0625bd6becd9f12c9e08057f5e4af30821594d35fb664ca not found: ID does not exist" Jul 28 17:54:38.016515 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.016472 2564 scope.go:117] "RemoveContainer" containerID="deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0" Jul 28 17:54:38.016740 ip-10-0-131-29 kubenswrapper[2564]: E0728 17:54:38.016718 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0\": container with ID starting with deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0 not found: ID does not exist" containerID="deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0" Jul 28 17:54:38.016868 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.016744 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0"} err="failed to get container status \"deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0\": rpc error: code = NotFound desc = could not find container \"deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0\": container with ID starting with deeffa3b3c21049a7cfc70d8c827053261a0e19ba9ff37894d1c79bcb61a96d0 not found: ID does not exist" Jul 28 17:54:38.020874 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.020845 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn"] Jul 28 17:54:38.024838 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:38.024809 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-cj2jn"] Jul 28 17:54:39.378291 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:54:39.378254 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" path="/var/lib/kubelet/pods/dd34e1a5-b0a3-4612-84bf-f211db085a4c/volumes" Jul 28 17:57:47.539815 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:57:47.539782 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:57:47.543622 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:57:47.543595 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 17:58:47.621907 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:58:47.621868 2564 scope.go:117] "RemoveContainer" containerID="650c98cc38182a0a8fb017cc4d509cd97e5e8c1b0bcd60dc13b204d8dfd15a3b" Jul 28 17:58:47.631724 ip-10-0-131-29 kubenswrapper[2564]: I0728 17:58:47.631700 2564 scope.go:117] "RemoveContainer" containerID="b209cf1702d42b6a40e50da1ee1a5233b41ff0cb055ef68e62ae61a82121eb8b" Jul 28 18:02:47.584847 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:02:47.584813 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:02:47.589273 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:02:47.589245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:03:12.921196 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:12.921102 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb"] Jul 28 18:03:12.930824 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:03:12.930752 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 18:03:12.930824 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:03:12.930824 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:03:12.930824 ip-10-0-131-29 kubenswrapper[2564]: > containerID="edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" cmd=["/bin/sleep","15"] Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:12.930841 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" message="" err=< Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:03:12.930854 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerName="main" containerID="cri-o://edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" Jul 28 18:03:12.931146 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:12.930904 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerName="main" containerID="cri-o://edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" gracePeriod=60 Jul 28 18:03:13.100154 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.100106 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused" Jul 28 18:03:13.295898 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.295865 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 18:03:13.463255 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463158 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-home\") pod \"9bf6b41c-19b6-4fce-a819-81229b680ca6\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " Jul 28 18:03:13.463255 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6b41c-19b6-4fce-a819-81229b680ca6-tls-certs\") pod \"9bf6b41c-19b6-4fce-a819-81229b680ca6\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " Jul 28 18:03:13.463255 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463218 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-model-cache\") pod \"9bf6b41c-19b6-4fce-a819-81229b680ca6\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " Jul 28 18:03:13.463816 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463303 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-tmp-dir\") pod \"9bf6b41c-19b6-4fce-a819-81229b680ca6\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " Jul 28 18:03:13.463816 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-dshm\") pod \"9bf6b41c-19b6-4fce-a819-81229b680ca6\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " Jul 28 18:03:13.463816 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7njp9\" (UniqueName: \"kubernetes.io/projected/9bf6b41c-19b6-4fce-a819-81229b680ca6-kube-api-access-7njp9\") pod \"9bf6b41c-19b6-4fce-a819-81229b680ca6\" (UID: \"9bf6b41c-19b6-4fce-a819-81229b680ca6\") " Jul 28 18:03:13.463816 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463545 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-home" (OuterVolumeSpecName: "home") pod "9bf6b41c-19b6-4fce-a819-81229b680ca6" (UID: "9bf6b41c-19b6-4fce-a819-81229b680ca6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:03:13.463816 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463743 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:03:13.464040 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463782 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bf6b41c-19b6-4fce-a819-81229b680ca6" (UID: "9bf6b41c-19b6-4fce-a819-81229b680ca6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:03:13.464040 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.463857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bf6b41c-19b6-4fce-a819-81229b680ca6" (UID: "9bf6b41c-19b6-4fce-a819-81229b680ca6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:03:13.465705 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.465675 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf6b41c-19b6-4fce-a819-81229b680ca6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bf6b41c-19b6-4fce-a819-81229b680ca6" (UID: "9bf6b41c-19b6-4fce-a819-81229b680ca6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:03:13.466113 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.466086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-dshm" (OuterVolumeSpecName: "dshm") pod "9bf6b41c-19b6-4fce-a819-81229b680ca6" (UID: "9bf6b41c-19b6-4fce-a819-81229b680ca6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:03:13.466113 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.466106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf6b41c-19b6-4fce-a819-81229b680ca6-kube-api-access-7njp9" (OuterVolumeSpecName: "kube-api-access-7njp9") pod "9bf6b41c-19b6-4fce-a819-81229b680ca6" (UID: "9bf6b41c-19b6-4fce-a819-81229b680ca6"). InnerVolumeSpecName "kube-api-access-7njp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:03:13.552360 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.552326 2564 generic.go:356] "Generic (PLEG): container finished" podID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerID="edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" exitCode=0 Jul 28 18:03:13.552547 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.552374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" event={"ID":"9bf6b41c-19b6-4fce-a819-81229b680ca6","Type":"ContainerDied","Data":"edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622"} Jul 28 18:03:13.552547 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.552416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" event={"ID":"9bf6b41c-19b6-4fce-a819-81229b680ca6","Type":"ContainerDied","Data":"ac61d22acf599799b3ec01c17134190252bd43eeddfdcbd25f463f8afaa4b457"} Jul 28 18:03:13.552547 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.552438 2564 scope.go:117] "RemoveContainer" containerID="edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" Jul 28 18:03:13.552547 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.552379 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb" Jul 28 18:03:13.563116 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.563093 2564 scope.go:117] "RemoveContainer" containerID="edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" Jul 28 18:03:13.563403 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:03:13.563383 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622\": container with ID starting with edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622 not found: ID does not exist" containerID="edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622" Jul 28 18:03:13.563498 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.563411 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622"} err="failed to get container status \"edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622\": rpc error: code = NotFound desc = could not find container \"edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622\": container with ID starting with edfbf0458d4fbd6938e01088bea17c70120ef2aa7f8a7e995d57c94fd7644622 not found: ID does not exist" Jul 28 18:03:13.565011 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.564984 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:03:13.565069 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.565017 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7njp9\" (UniqueName: \"kubernetes.io/projected/9bf6b41c-19b6-4fce-a819-81229b680ca6-kube-api-access-7njp9\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:03:13.565069 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.565030 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6b41c-19b6-4fce-a819-81229b680ca6-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:03:13.565069 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.565061 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:03:13.565167 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.565073 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6b41c-19b6-4fce-a819-81229b680ca6-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:03:13.581073 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.581026 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb"] Jul 28 18:03:13.583393 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:13.583365 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb"] Jul 28 18:03:15.377491 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:15.377454 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" path="/var/lib/kubelet/pods/9bf6b41c-19b6-4fce-a819-81229b680ca6/volumes" Jul 28 18:03:36.419997 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.419953 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52"] Jul 28 18:03:36.420426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420350 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="main" Jul 28 18:03:36.420426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420362 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="main" Jul 28 18:03:36.420426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420381 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerName="main" Jul 28 18:03:36.420426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420387 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerName="main" Jul 28 18:03:36.420426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420396 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="storage-initializer" Jul 28 18:03:36.420426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420402 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="storage-initializer" Jul 28 18:03:36.420647 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420466 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf6b41c-19b6-4fce-a819-81229b680ca6" containerName="main" Jul 28 18:03:36.420647 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.420475 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd34e1a5-b0a3-4612-84bf-f211db085a4c" containerName="main" Jul 28 18:03:36.423596 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.423568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.429497 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.429431 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-mq9rw\"" Jul 28 18:03:36.429497 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.429437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\"" Jul 28 18:03:36.429777 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.429437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\"" Jul 28 18:03:36.429777 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.429443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\"" Jul 28 18:03:36.439216 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.439173 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52"] Jul 28 18:03:36.478943 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.478905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.478943 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.478944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctb6\" (UniqueName: \"kubernetes.io/projected/42c9131d-8b0f-492c-961b-ad36b6a51281-kube-api-access-kctb6\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.479185 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.478965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.479185 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.479071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.479185 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.479142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c9131d-8b0f-492c-961b-ad36b6a51281-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.479185 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.479170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.579950 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.579917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c9131d-8b0f-492c-961b-ad36b6a51281-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.579950 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.579956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580208 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.579992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580208 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.580012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctb6\" (UniqueName: \"kubernetes.io/projected/42c9131d-8b0f-492c-961b-ad36b6a51281-kube-api-access-kctb6\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580208 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.580135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580208 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.580199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580448 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.580364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580448 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.580404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.580525 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.580479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.582605 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.582582 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.582723 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.582634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c9131d-8b0f-492c-961b-ad36b6a51281-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.593453 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.593423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctb6\" (UniqueName: \"kubernetes.io/projected/42c9131d-8b0f-492c-961b-ad36b6a51281-kube-api-access-kctb6\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.734826 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.734697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:36.886330 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:36.886299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52"] Jul 28 18:03:36.888697 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:03:36.888663 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c9131d_8b0f_492c_961b_ad36b6a51281.slice/crio-e1371de746f55f9da647e421793925a67ccd9ac36cd539766a61c12ce7f663b1 WatchSource:0}: Error finding container e1371de746f55f9da647e421793925a67ccd9ac36cd539766a61c12ce7f663b1: Status 404 returned error can't find the container with id e1371de746f55f9da647e421793925a67ccd9ac36cd539766a61c12ce7f663b1 Jul 28 18:03:37.672689 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:37.672643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" event={"ID":"42c9131d-8b0f-492c-961b-ad36b6a51281","Type":"ContainerStarted","Data":"0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055"} Jul 28 18:03:37.673131 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:37.672712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" event={"ID":"42c9131d-8b0f-492c-961b-ad36b6a51281","Type":"ContainerStarted","Data":"e1371de746f55f9da647e421793925a67ccd9ac36cd539766a61c12ce7f663b1"} Jul 28 18:03:37.702662 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:37.702595 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" podStartSLOduration=1.702573473 podStartE2EDuration="1.702573473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:37.702360404 +0000 UTC m=+2450.854860581" watchObservedRunningTime="2026-07-28 18:03:37.702573473 +0000 UTC m=+2450.855073651" Jul 28 18:03:37.735269 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:37.735226 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:46.734848 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:46.734808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:46.747530 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:46.747496 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:03:47.733785 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:03:47.733740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:07:47.619195 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:07:47.619161 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:07:47.623538 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:07:47.623505 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:12:47.652105 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:12:47.652071 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:12:47.658268 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:12:47.658237 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:17:47.690958 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:17:47.690925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:17:47.698882 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:17:47.698850 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:18:40.099277 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.099241 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52"] Jul 28 18:18:40.108456 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:18:40.108417 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 18:18:40.108456 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:18:40.108456 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:18:40.108456 ip-10-0-131-29 kubenswrapper[2564]: > containerID="0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" cmd=["/bin/sleep","15"] Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.108470 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" message="" err=< Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:18:40.108487 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerName="main" containerID="cri-o://0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" Jul 28 18:18:40.108723 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.108537 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerName="main" containerID="cri-o://0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" gracePeriod=60 Jul 28 18:18:40.722745 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.722460 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused" Jul 28 18:18:40.862166 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.862138 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:18:40.978947 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.978844 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-home\") pod \"42c9131d-8b0f-492c-961b-ad36b6a51281\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " Jul 28 18:18:40.978947 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.978898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-tmp-dir\") pod \"42c9131d-8b0f-492c-961b-ad36b6a51281\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " Jul 28 18:18:40.979188 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.978962 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c9131d-8b0f-492c-961b-ad36b6a51281-tls-certs\") pod \"42c9131d-8b0f-492c-961b-ad36b6a51281\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " Jul 28 18:18:40.979188 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.978996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-model-cache\") pod \"42c9131d-8b0f-492c-961b-ad36b6a51281\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " Jul 28 18:18:40.979188 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979039 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kctb6\" (UniqueName: \"kubernetes.io/projected/42c9131d-8b0f-492c-961b-ad36b6a51281-kube-api-access-kctb6\") pod \"42c9131d-8b0f-492c-961b-ad36b6a51281\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " Jul 28 18:18:40.979188 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979062 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-dshm\") pod \"42c9131d-8b0f-492c-961b-ad36b6a51281\" (UID: \"42c9131d-8b0f-492c-961b-ad36b6a51281\") " Jul 28 18:18:40.979360 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979180 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-home" (OuterVolumeSpecName: "home") pod "42c9131d-8b0f-492c-961b-ad36b6a51281" (UID: "42c9131d-8b0f-492c-961b-ad36b6a51281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:18:40.979360 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979236 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42c9131d-8b0f-492c-961b-ad36b6a51281" (UID: "42c9131d-8b0f-492c-961b-ad36b6a51281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:18:40.979360 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979262 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-model-cache" (OuterVolumeSpecName: "model-cache") pod "42c9131d-8b0f-492c-961b-ad36b6a51281" (UID: "42c9131d-8b0f-492c-961b-ad36b6a51281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:18:40.979512 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979413 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:18:40.979512 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979433 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:18:40.979512 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.979446 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:18:40.981299 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.981270 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c9131d-8b0f-492c-961b-ad36b6a51281-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42c9131d-8b0f-492c-961b-ad36b6a51281" (UID: "42c9131d-8b0f-492c-961b-ad36b6a51281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:18:40.981299 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.981285 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-dshm" (OuterVolumeSpecName: "dshm") pod "42c9131d-8b0f-492c-961b-ad36b6a51281" (UID: "42c9131d-8b0f-492c-961b-ad36b6a51281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:18:40.981455 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:40.981299 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c9131d-8b0f-492c-961b-ad36b6a51281-kube-api-access-kctb6" (OuterVolumeSpecName: "kube-api-access-kctb6") pod "42c9131d-8b0f-492c-961b-ad36b6a51281" (UID: "42c9131d-8b0f-492c-961b-ad36b6a51281"). InnerVolumeSpecName "kube-api-access-kctb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:18:41.080971 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.080935 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c9131d-8b0f-492c-961b-ad36b6a51281-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:18:41.080971 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.080966 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kctb6\" (UniqueName: \"kubernetes.io/projected/42c9131d-8b0f-492c-961b-ad36b6a51281-kube-api-access-kctb6\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:18:41.081206 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.080982 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c9131d-8b0f-492c-961b-ad36b6a51281-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:18:41.190101 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.190067 2564 generic.go:356] "Generic (PLEG): container finished" podID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerID="0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" exitCode=0 Jul 28 18:18:41.190560 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.190107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" event={"ID":"42c9131d-8b0f-492c-961b-ad36b6a51281","Type":"ContainerDied","Data":"0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055"} Jul 28 18:18:41.190560 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.190139 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" Jul 28 18:18:41.190560 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.190174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52" event={"ID":"42c9131d-8b0f-492c-961b-ad36b6a51281","Type":"ContainerDied","Data":"e1371de746f55f9da647e421793925a67ccd9ac36cd539766a61c12ce7f663b1"} Jul 28 18:18:41.190560 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.190198 2564 scope.go:117] "RemoveContainer" containerID="0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" Jul 28 18:18:41.199458 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.199434 2564 scope.go:117] "RemoveContainer" containerID="0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" Jul 28 18:18:41.199822 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:18:41.199802 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055\": container with ID starting with 0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055 not found: ID does not exist" containerID="0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055" Jul 28 18:18:41.199864 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.199832 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055"} err="failed to get container status \"0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055\": rpc error: code = NotFound desc = could not find container \"0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055\": container with ID starting with 0db22f5866665c65610d2dad839f422d5eddf4b3e59a40c0fd893db9bafe2055 not found: ID does not exist" Jul 28 18:18:41.217033 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.216999 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52"] Jul 28 18:18:41.228408 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.228376 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52"] Jul 28 18:18:41.377710 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:18:41.377678 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" path="/var/lib/kubelet/pods/42c9131d-8b0f-492c-961b-ad36b6a51281/volumes" Jul 28 18:19:22.566843 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.566799 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"] Jul 28 18:19:22.567452 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.567364 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerName="main" Jul 28 18:19:22.567452 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.567383 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerName="main" Jul 28 18:19:22.567569 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.567490 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c9131d-8b0f-492c-961b-ad36b6a51281" containerName="main" Jul 28 18:19:22.570677 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.570654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.575304 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.575280 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-v865s\"" Jul 28 18:19:22.575459 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.575367 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\"" Jul 28 18:19:22.576536 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.576511 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-k2mvb\"" Jul 28 18:19:22.577618 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.577425 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\"" Jul 28 18:19:22.577618 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.577429 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\"" Jul 28 18:19:22.589215 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.589180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"] Jul 28 18:19:22.645232 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.645194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.645429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.645246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968gf\" (UniqueName: \"kubernetes.io/projected/d48f36b1-1351-4dc6-ac22-76261202560a-kube-api-access-968gf\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.645429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.645302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48f36b1-1351-4dc6-ac22-76261202560a-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.645429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.645327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.645429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.645398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.645429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.645426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746076 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746076 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746345 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746345 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968gf\" (UniqueName: \"kubernetes.io/projected/d48f36b1-1351-4dc6-ac22-76261202560a-kube-api-access-968gf\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746345 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48f36b1-1351-4dc6-ac22-76261202560a-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746345 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746555 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746599 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.746643 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.746595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.748547 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.748523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.748644 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.748571 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48f36b1-1351-4dc6-ac22-76261202560a-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.757913 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.757882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968gf\" (UniqueName: \"kubernetes.io/projected/d48f36b1-1351-4dc6-ac22-76261202560a-kube-api-access-968gf\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:22.881645 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:22.881375 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:23.039735 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:23.039701 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"] Jul 28 18:19:23.043949 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:19:23.043918 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48f36b1_1351_4dc6_ac22_76261202560a.slice/crio-9cecbdf2f838b48ce68fce43632b1ec31b09242a18e38033e47ab8a2272f18ee WatchSource:0}: Error finding container 9cecbdf2f838b48ce68fce43632b1ec31b09242a18e38033e47ab8a2272f18ee: Status 404 returned error can't find the container with id 9cecbdf2f838b48ce68fce43632b1ec31b09242a18e38033e47ab8a2272f18ee Jul 28 18:19:23.394397 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:23.394355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d48f36b1-1351-4dc6-ac22-76261202560a","Type":"ContainerStarted","Data":"6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412"} Jul 28 18:19:23.394605 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:23.394416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d48f36b1-1351-4dc6-ac22-76261202560a","Type":"ContainerStarted","Data":"9cecbdf2f838b48ce68fce43632b1ec31b09242a18e38033e47ab8a2272f18ee"} Jul 28 18:19:23.419203 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:23.419145 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podStartSLOduration=1.419129075 podStartE2EDuration="1.419129075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:23.418751733 +0000 UTC m=+3396.571251910" watchObservedRunningTime="2026-07-28 18:19:23.419129075 +0000 UTC m=+3396.571629252" Jul 28 18:19:23.882169 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:23.882130 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:32.881986 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:32.881942 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:32.894994 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:32.894964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:19:33.456139 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:19:33.456106 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:22:47.722905 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:22:47.722791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:22:47.731075 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:22:47.731050 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:23:34.199878 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.199844 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r"] Jul 28 18:23:34.203357 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.203337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:34.207076 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.207053 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-xgkv6\"" Jul 28 18:23:34.208366 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.208350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\"" Jul 28 18:23:34.208453 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.208432 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\"" Jul 28 18:23:34.285959 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.285923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r"] Jul 28 18:23:34.327003 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.326965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8v2\" (UniqueName: \"kubernetes.io/projected/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65-kube-api-access-tc8v2\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:34.428326 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.428283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8v2\" (UniqueName: \"kubernetes.io/projected/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65-kube-api-access-tc8v2\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:34.439039 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:34.439011 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8v2\" (UniqueName: \"kubernetes.io/projected/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65-kube-api-access-tc8v2\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.648079 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.648039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.655060 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.655030 2564 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.749206 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.749161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.749395 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.749288 2564 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.753109 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.753084 2564 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") DevicePath \"csi-28549b9ed4a63d84e88479665e8b9fe123f3d5e7c0289c2d4e2be9a2dcbc5781\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.936447 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.936409 2564 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4b84170d5113e5de0a67610f5328f5c4f158d139382bdd67d29c04dd9506ed15/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:36.953062 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:36.953031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2e-pvc-model-download-gfq5r\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:37.213237 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:37.213125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:23:37.369248 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:37.369212 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r"] Jul 28 18:23:37.372860 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:23:37.372829 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2dd99e0_1ac9_4ef5_bda6_586a08f15a65.slice/crio-90409e47d3549790ef62f983a5076027846e883a9912a9d84c085a5be4ed50c4 WatchSource:0}: Error finding container 90409e47d3549790ef62f983a5076027846e883a9912a9d84c085a5be4ed50c4: Status 404 returned error can't find the container with id 90409e47d3549790ef62f983a5076027846e883a9912a9d84c085a5be4ed50c4 Jul 28 18:23:37.669483 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:37.669444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" event={"ID":"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65","Type":"ContainerStarted","Data":"1c366871f0022f4d9c4e1cc4172af9c5953d4fa38bbccaf26f02fc53517ab0bf"} Jul 28 18:23:37.669483 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:37.669490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" event={"ID":"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65","Type":"ContainerStarted","Data":"90409e47d3549790ef62f983a5076027846e883a9912a9d84c085a5be4ed50c4"} Jul 28 18:23:37.702088 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:23:37.702022 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" podStartSLOduration=6.702005114 podStartE2EDuration="6.702005114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:23:37.700741227 +0000 UTC m=+3650.853241406" watchObservedRunningTime="2026-07-28 18:23:37.702005114 +0000 UTC m=+3650.854505290" Jul 28 18:24:04.816702 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:04.816597 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" containerID="1c366871f0022f4d9c4e1cc4172af9c5953d4fa38bbccaf26f02fc53517ab0bf" exitCode=0 Jul 28 18:24:04.816702 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:04.816644 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" event={"ID":"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65","Type":"ContainerDied","Data":"1c366871f0022f4d9c4e1cc4172af9c5953d4fa38bbccaf26f02fc53517ab0bf"} Jul 28 18:24:05.983290 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:05.983263 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:24:06.125741 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.125700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc8v2\" (UniqueName: \"kubernetes.io/projected/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65-kube-api-access-tc8v2\") pod \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " Jul 28 18:24:06.125934 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.125881 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\" (UID: \"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65\") " Jul 28 18:24:06.127999 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.127967 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65-kube-api-access-tc8v2" (OuterVolumeSpecName: "kube-api-access-tc8v2") pod "e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" (UID: "e2dd99e0-1ac9-4ef5-bda6-586a08f15a65"). InnerVolumeSpecName "kube-api-access-tc8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:24:06.129738 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.129718 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1" (OuterVolumeSpecName: "model-storage") pod "e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" (UID: "e2dd99e0-1ac9-4ef5-bda6-586a08f15a65"). InnerVolumeSpecName "pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227". PluginName "kubernetes.io/csi", VolumeGIDValue "" Jul 28 18:24:06.226777 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.226715 2564 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") on node \"ip-10-0-131-29.ec2.internal\" " Jul 28 18:24:06.226777 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.226749 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc8v2\" (UniqueName: \"kubernetes.io/projected/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65-kube-api-access-tc8v2\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:24:06.342628 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.342602 2564 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1") on node "ip-10-0-131-29.ec2.internal" Jul 28 18:24:06.429356 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.429258 2564 reconciler_common.go:299] "Volume detached for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"csi-28549b9ed4a63d84e88479665e8b9fe123f3d5e7c0289c2d4e2be9a2dcbc5781\"" Jul 28 18:24:06.827663 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.827635 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" Jul 28 18:24:06.827663 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.827647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r" event={"ID":"e2dd99e0-1ac9-4ef5-bda6-586a08f15a65","Type":"ContainerDied","Data":"90409e47d3549790ef62f983a5076027846e883a9912a9d84c085a5be4ed50c4"} Jul 28 18:24:06.827999 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:06.827685 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90409e47d3549790ef62f983a5076027846e883a9912a9d84c085a5be4ed50c4" Jul 28 18:24:25.206836 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.206725 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"] Jul 28 18:24:25.207331 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.207312 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" containerName="storage-initializer" Jul 28 18:24:25.207404 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.207334 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" containerName="storage-initializer" Jul 28 18:24:25.207459 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.207438 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" containerName="storage-initializer" Jul 28 18:24:25.210689 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.210668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.217894 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.217866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\"" Jul 28 18:24:25.218003 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.217947 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\"" Jul 28 18:24:25.218003 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.217979 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\"" Jul 28 18:24:25.218075 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.217871 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-xgkv6\"" Jul 28 18:24:25.218363 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.218342 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zcd55\"" Jul 28 18:24:25.237174 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.237136 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"] Jul 28 18:24:25.300295 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.300249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.300503 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.300317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74fn\" (UniqueName: \"kubernetes.io/projected/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-kube-api-access-x74fn\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.300503 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.300394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.300503 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.300424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.300503 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.300484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.300752 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.300506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401381 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401381 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401635 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401635 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74fn\" (UniqueName: \"kubernetes.io/projected/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-kube-api-access-x74fn\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401635 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401635 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401882 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.401943 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401878 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.402003 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.401936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.403896 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.403874 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.403896 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.403883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:25.433958 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:25.433923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74fn\" (UniqueName: \"kubernetes.io/projected/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-kube-api-access-x74fn\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:28.933177 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:28.933133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:28.940253 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:28.940221 2564 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.034584 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.034542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.034791 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.034676 2564 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.039916 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.039893 2564 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") DevicePath \"csi-28549b9ed4a63d84e88479665e8b9fe123f3d5e7c0289c2d4e2be9a2dcbc5781\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.129610 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.129576 2564 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4b84170d5113e5de0a67610f5328f5c4f158d139382bdd67d29c04dd9506ed15/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.142204 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.142173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.421796 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.421736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:29.571270 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.571242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"] Jul 28 18:24:29.573816 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:24:29.573784 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1aa2b26_ff50_41bc_bf6c_44d3428d69b3.slice/crio-612d790e009c204f381747561672f165e0f796dd5b20fa0e4f6e59ea6139ddd8 WatchSource:0}: Error finding container 612d790e009c204f381747561672f165e0f796dd5b20fa0e4f6e59ea6139ddd8: Status 404 returned error can't find the container with id 612d790e009c204f381747561672f165e0f796dd5b20fa0e4f6e59ea6139ddd8 Jul 28 18:24:29.944904 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.944862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3","Type":"ContainerStarted","Data":"dcb153c9a763e94f3838a953d73bc84f42ac5a74ec5a723de3e9178978415232"} Jul 28 18:24:29.945303 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.944929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3","Type":"ContainerStarted","Data":"612d790e009c204f381747561672f165e0f796dd5b20fa0e4f6e59ea6139ddd8"} Jul 28 18:24:29.965968 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:29.965908 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=4.965891699 podStartE2EDuration="4.965891699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:29.963841421 +0000 UTC m=+3703.116341625" watchObservedRunningTime="2026-07-28 18:24:29.965891699 +0000 UTC m=+3703.118391876" Jul 28 18:24:30.422477 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:30.422429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:39.422478 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:39.422419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:24:39.424131 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:39.424094 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:24:49.422725 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:49.422670 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:24:59.423004 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:24:59.422954 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:25:09.422707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:25:09.422641 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:25:19.423271 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:25:19.423221 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:25:29.422774 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:25:29.422712 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:25:39.423034 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:25:39.422954 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:25:49.422702 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:25:49.422656 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:25:59.423238 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:25:59.423190 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:26:09.433815 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:09.433751 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:26:09.442032 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:09.442000 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:26:38.067854 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:38.067815 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"] Jul 28 18:26:53.091086 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.091038 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" containerID="cri-o://dcb153c9a763e94f3838a953d73bc84f42ac5a74ec5a723de3e9178978415232" gracePeriod=45 Jul 28 18:26:53.435198 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.435100 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused" Jul 28 18:26:53.725355 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.725323 2564 generic.go:356] "Generic (PLEG): container finished" podID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerID="dcb153c9a763e94f3838a953d73bc84f42ac5a74ec5a723de3e9178978415232" exitCode=0 Jul 28 18:26:53.725551 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.725399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3","Type":"ContainerDied","Data":"dcb153c9a763e94f3838a953d73bc84f42ac5a74ec5a723de3e9178978415232"} Jul 28 18:26:53.845102 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.845069 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:26:53.926322 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926279 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-dshm\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.926519 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926347 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x74fn\" (UniqueName: \"kubernetes.io/projected/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-kube-api-access-x74fn\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.926519 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tmp-dir\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.926519 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tls-certs\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.926519 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-model-cache\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.926786 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.926786 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926693 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:53.926786 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.926735 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-home\") pod \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\" (UID: \"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3\") " Jul 28 18:26:53.927164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.927110 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:26:53.927434 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.927376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-home" (OuterVolumeSpecName: "home") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:53.928729 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.928701 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-dshm" (OuterVolumeSpecName: "dshm") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:53.929081 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.929056 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:26:53.929204 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.929185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-kube-api-access-x74fn" (OuterVolumeSpecName: "kube-api-access-x74fn") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "kube-api-access-x74fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:26:53.929548 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.929530 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1" (OuterVolumeSpecName: "kserve-pvc-source") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227". PluginName "kubernetes.io/csi", VolumeGIDValue "" Jul 28 18:26:53.938121 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:53.938089 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" (UID: "e1aa2b26-ff50-41bc-bf6c-44d3428d69b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:54.028164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.028074 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:26:54.028164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.028105 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:26:54.028164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.028117 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74fn\" (UniqueName: \"kubernetes.io/projected/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-kube-api-access-x74fn\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:26:54.028164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.028127 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:26:54.028164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.028136 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:26:54.028164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.028164 2564 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") on node \"ip-10-0-131-29.ec2.internal\" " Jul 28 18:26:54.056248 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.056228 2564 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1") on node "ip-10-0-131-29.ec2.internal" Jul 28 18:26:54.129526 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.129475 2564 reconciler_common.go:299] "Volume detached for volume \"pvc-a7b28789-ec5e-469c-8226-a8c9a0c9d227\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019dba429c9dda1b1\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"csi-28549b9ed4a63d84e88479665e8b9fe123f3d5e7c0289c2d4e2be9a2dcbc5781\"" Jul 28 18:26:54.732097 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.732057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"e1aa2b26-ff50-41bc-bf6c-44d3428d69b3","Type":"ContainerDied","Data":"612d790e009c204f381747561672f165e0f796dd5b20fa0e4f6e59ea6139ddd8"} Jul 28 18:26:54.732097 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.732079 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" Jul 28 18:26:54.732310 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.732143 2564 scope.go:117] "RemoveContainer" containerID="dcb153c9a763e94f3838a953d73bc84f42ac5a74ec5a723de3e9178978415232" Jul 28 18:26:54.757278 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.757245 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"] Jul 28 18:26:54.764550 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:54.764520 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"] Jul 28 18:26:55.377122 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:26:55.377084 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" path="/var/lib/kubelet/pods/e1aa2b26-ff50-41bc-bf6c-44d3428d69b3/volumes" Jul 28 18:27:03.021928 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:03.021888 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r"] Jul 28 18:27:03.025956 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:03.025920 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-gfq5r"] Jul 28 18:27:03.377081 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:03.377046 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2dd99e0-1ac9-4ef5-bda6-586a08f15a65" path="/var/lib/kubelet/pods/e2dd99e0-1ac9-4ef5-bda6-586a08f15a65/volumes" Jul 28 18:27:46.668704 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.668668 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv"] Jul 28 18:27:46.669170 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.669084 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" Jul 28 18:27:46.669170 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.669096 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" Jul 28 18:27:46.669170 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.669163 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1aa2b26-ff50-41bc-bf6c-44d3428d69b3" containerName="main" Jul 28 18:27:46.672859 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.672834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.676637 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.676596 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\"" Jul 28 18:27:46.676901 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.676883 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\"" Jul 28 18:27:46.676972 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.676901 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-4gvhr\"" Jul 28 18:27:46.676972 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.676883 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\"" Jul 28 18:27:46.677826 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.677808 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\"" Jul 28 18:27:46.702352 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.702315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv"] Jul 28 18:27:46.795784 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.795708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-home\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.795784 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.795751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.796014 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.795817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.796014 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.795834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.796014 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.795919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.796014 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.795957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgm64\" (UniqueName: \"kubernetes.io/projected/b421f9fb-5d85-4ff9-a007-eec4522fc072-kube-api-access-cgm64\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.796146 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.796028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b421f9fb-5d85-4ff9-a007-eec4522fc072-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897112 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897112 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgm64\" (UniqueName: \"kubernetes.io/projected/b421f9fb-5d85-4ff9-a007-eec4522fc072-kube-api-access-cgm64\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897364 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b421f9fb-5d85-4ff9-a007-eec4522fc072-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897364 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-home\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897364 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897364 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897364 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897698 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-home\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897805 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897805 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.897896 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.897785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.899479 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.899456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b421f9fb-5d85-4ff9-a007-eec4522fc072-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.899787 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.899746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b421f9fb-5d85-4ff9-a007-eec4522fc072-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.906300 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.906267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgm64\" (UniqueName: \"kubernetes.io/projected/b421f9fb-5d85-4ff9-a007-eec4522fc072-kube-api-access-cgm64\") pod \"stop-feature-test-kserve-65499d9cfd-rmdqv\" (UID: \"b421f9fb-5d85-4ff9-a007-eec4522fc072\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:46.983831 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:46.983700 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" Jul 28 18:27:47.146519 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:47.146488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv"] Jul 28 18:27:47.147577 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:27:47.147544 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb421f9fb_5d85_4ff9_a007_eec4522fc072.slice/crio-76008c04f807594cb68815fad6ac2036a95aa9881618b39e4d06523a80a46f65 WatchSource:0}: Error finding container 76008c04f807594cb68815fad6ac2036a95aa9881618b39e4d06523a80a46f65: Status 404 returned error can't find the container with id 76008c04f807594cb68815fad6ac2036a95aa9881618b39e4d06523a80a46f65 Jul 28 18:27:47.757733 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:47.757626 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:27:47.771716 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:47.767903 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:27:48.007016 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:48.006973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" event={"ID":"b421f9fb-5d85-4ff9-a007-eec4522fc072","Type":"ContainerStarted","Data":"a840f063c0004cfba19ea532c817a7e10fc99145826edd164f6689b1a5fb5527"} Jul 28 18:27:48.007227 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:27:48.007074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-rmdqv" event={"ID":"b421f9fb-5d85-4ff9-a007-eec4522fc072","Type":"ContainerStarted","Data":"76008c04f807594cb68815fad6ac2036a95aa9881618b39e4d06523a80a46f65"} Jul 28 18:29:47.901609 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:29:47.901573 2564 scope.go:117] "RemoveContainer" containerID="1c366871f0022f4d9c4e1cc4172af9c5953d4fa38bbccaf26f02fc53517ab0bf" Jul 28 18:32:47.790537 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:32:47.790432 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:32:47.802028 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:32:47.802002 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:34:13.241541 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.241501 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"] Jul 28 18:34:13.252096 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:34:13.252058 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 18:34:13.252096 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:34:13.252096 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:34:13.252096 ip-10-0-131-29 kubenswrapper[2564]: > containerID="6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" cmd=["/bin/sleep","15"] Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.252111 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" message="" err=< Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:34:13.252124 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" containerName="main" containerID="cri-o://6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" Jul 28 18:34:13.252447 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.252179 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" containerName="main" containerID="cri-o://6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" gracePeriod=60 Jul 28 18:34:13.445724 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.445679 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused" Jul 28 18:34:13.613426 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.613399 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:34:13.778536 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778440 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-tmp-dir\") pod \"d48f36b1-1351-4dc6-ac22-76261202560a\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " Jul 28 18:34:13.778707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778542 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-model-cache\") pod \"d48f36b1-1351-4dc6-ac22-76261202560a\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " Jul 28 18:34:13.778707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778582 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-dshm\") pod \"d48f36b1-1351-4dc6-ac22-76261202560a\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " Jul 28 18:34:13.778707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-home\") pod \"d48f36b1-1351-4dc6-ac22-76261202560a\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " Jul 28 18:34:13.778707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-968gf\" (UniqueName: \"kubernetes.io/projected/d48f36b1-1351-4dc6-ac22-76261202560a-kube-api-access-968gf\") pod \"d48f36b1-1351-4dc6-ac22-76261202560a\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " Jul 28 18:34:13.778707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48f36b1-1351-4dc6-ac22-76261202560a-tls-certs\") pod \"d48f36b1-1351-4dc6-ac22-76261202560a\" (UID: \"d48f36b1-1351-4dc6-ac22-76261202560a\") " Jul 28 18:34:13.779028 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778775 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d48f36b1-1351-4dc6-ac22-76261202560a" (UID: "d48f36b1-1351-4dc6-ac22-76261202560a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:13.779028 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.778984 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-home" (OuterVolumeSpecName: "home") pod "d48f36b1-1351-4dc6-ac22-76261202560a" (UID: "d48f36b1-1351-4dc6-ac22-76261202560a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:13.779028 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.779012 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:34:13.779028 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.779017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d48f36b1-1351-4dc6-ac22-76261202560a" (UID: "d48f36b1-1351-4dc6-ac22-76261202560a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:13.781153 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.781126 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48f36b1-1351-4dc6-ac22-76261202560a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d48f36b1-1351-4dc6-ac22-76261202560a" (UID: "d48f36b1-1351-4dc6-ac22-76261202560a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:34:13.781288 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.781269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-dshm" (OuterVolumeSpecName: "dshm") pod "d48f36b1-1351-4dc6-ac22-76261202560a" (UID: "d48f36b1-1351-4dc6-ac22-76261202560a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:13.781350 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.781311 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48f36b1-1351-4dc6-ac22-76261202560a-kube-api-access-968gf" (OuterVolumeSpecName: "kube-api-access-968gf") pod "d48f36b1-1351-4dc6-ac22-76261202560a" (UID: "d48f36b1-1351-4dc6-ac22-76261202560a"). InnerVolumeSpecName "kube-api-access-968gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:34:13.880110 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.880073 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:34:13.880110 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.880101 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:34:13.880110 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.880109 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48f36b1-1351-4dc6-ac22-76261202560a-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:34:13.880110 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.880118 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968gf\" (UniqueName: \"kubernetes.io/projected/d48f36b1-1351-4dc6-ac22-76261202560a-kube-api-access-968gf\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:34:13.880422 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.880127 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48f36b1-1351-4dc6-ac22-76261202560a-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:34:13.958047 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.958011 2564 generic.go:356] "Generic (PLEG): container finished" podID="d48f36b1-1351-4dc6-ac22-76261202560a" containerID="6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" exitCode=0 Jul 28 18:34:13.958226 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.958058 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" Jul 28 18:34:13.958226 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.958095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d48f36b1-1351-4dc6-ac22-76261202560a","Type":"ContainerDied","Data":"6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412"} Jul 28 18:34:13.958226 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.958145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d48f36b1-1351-4dc6-ac22-76261202560a","Type":"ContainerDied","Data":"9cecbdf2f838b48ce68fce43632b1ec31b09242a18e38033e47ab8a2272f18ee"} Jul 28 18:34:13.958226 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.958167 2564 scope.go:117] "RemoveContainer" containerID="6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" Jul 28 18:34:13.970187 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.970171 2564 scope.go:117] "RemoveContainer" containerID="6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" Jul 28 18:34:13.970449 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:34:13.970432 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412\": container with ID starting with 6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412 not found: ID does not exist" containerID="6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412" Jul 28 18:34:13.970512 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.970458 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412"} err="failed to get container status \"6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412\": rpc error: code = NotFound desc = could not find container \"6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412\": container with ID starting with 6d0b38e2f45b0045993cebc9687c4bb3072b6028af0ab0e879a9c067b7339412 not found: ID does not exist" Jul 28 18:34:13.983691 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.983666 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"] Jul 28 18:34:13.987975 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:13.987950 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"] Jul 28 18:34:15.377150 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:15.377117 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" path="/var/lib/kubelet/pods/d48f36b1-1351-4dc6-ac22-76261202560a/volumes" Jul 28 18:34:46.967649 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.967553 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"] Jul 28 18:34:46.968154 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.968001 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" containerName="main" Jul 28 18:34:46.968154 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.968015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" containerName="main" Jul 28 18:34:46.968154 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.968092 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48f36b1-1351-4dc6-ac22-76261202560a" containerName="main" Jul 28 18:34:46.971252 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.971234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:46.974050 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.974027 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\"" Jul 28 18:34:46.975483 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.975463 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\"" Jul 28 18:34:46.975594 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.975463 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-pfdqg\"" Jul 28 18:34:46.975594 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.975463 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\"" Jul 28 18:34:46.975594 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.975524 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-qr8kw\"" Jul 28 18:34:46.981090 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:46.981066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"] Jul 28 18:34:47.096392 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.096346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.096592 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.096420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.096592 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.096493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.096592 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.096523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.096592 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.096577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthvz\" (UniqueName: \"kubernetes.io/projected/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-kube-api-access-jthvz\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.096790 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.096607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198224 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198224 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198509 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthvz\" (UniqueName: \"kubernetes.io/projected/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-kube-api-access-jthvz\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198509 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198509 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198752 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198752 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198752 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.198919 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.198864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.200804 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.200748 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.201030 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.201005 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.207725 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.207691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthvz\" (UniqueName: \"kubernetes.io/projected/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-kube-api-access-jthvz\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.283801 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.283690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:47.432321 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:47.432295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"] Jul 28 18:34:47.434796 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:34:47.434749 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2bfc6e_a038_494f_a1f4_ea93137b70ea.slice/crio-ef4ccd371666ea2d8c9dff51ffc735b587811d64e9cc05b0c091284c190fbc2f WatchSource:0}: Error finding container ef4ccd371666ea2d8c9dff51ffc735b587811d64e9cc05b0c091284c190fbc2f: Status 404 returned error can't find the container with id ef4ccd371666ea2d8c9dff51ffc735b587811d64e9cc05b0c091284c190fbc2f Jul 28 18:34:48.131790 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:48.131731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4e2bfc6e-a038-494f-a1f4-ea93137b70ea","Type":"ContainerStarted","Data":"55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac"} Jul 28 18:34:48.132167 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:48.131800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4e2bfc6e-a038-494f-a1f4-ea93137b70ea","Type":"ContainerStarted","Data":"ef4ccd371666ea2d8c9dff51ffc735b587811d64e9cc05b0c091284c190fbc2f"} Jul 28 18:34:48.153395 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:48.153338 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podStartSLOduration=2.153322414 podStartE2EDuration="2.153322414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:48.152630442 +0000 UTC m=+4321.305130665" watchObservedRunningTime="2026-07-28 18:34:48.153322414 +0000 UTC m=+4321.305822590" Jul 28 18:34:48.284803 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:48.284668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:57.284150 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:57.284107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:57.296864 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:57.296836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:34:58.193012 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:34:58.192978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:37:47.828871 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:37:47.828841 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:37:47.836245 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:37:47.836212 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:42:47.861517 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:42:47.861407 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:42:47.870323 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:42:47.870292 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:47:47.892710 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:47:47.892599 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:47:47.902930 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:47:47.902900 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:49:07.937712 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.937676 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n"] Jul 28 18:49:07.941076 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.941056 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:07.944446 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.944425 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\"" Jul 28 18:49:07.944983 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.944966 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\"" Jul 28 18:49:07.945086 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.945057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qs25j\"" Jul 28 18:49:07.945159 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.945091 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\"" Jul 28 18:49:07.945628 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.945611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\"" Jul 28 18:49:07.952872 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:07.952853 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n"] Jul 28 18:49:08.040522 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9cz\" (UniqueName: \"kubernetes.io/projected/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kube-api-access-rw9cz\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.040737 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-dshm\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.040737 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-model-cache\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.040737 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kserve-provision-location\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.040737 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tmp-dir\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.040941 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tls-certs\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.040941 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.040807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-home\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.141622 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tmp-dir\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.141622 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tls-certs\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.141890 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-home\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.141890 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9cz\" (UniqueName: \"kubernetes.io/projected/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kube-api-access-rw9cz\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.141890 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-dshm\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.141890 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-model-cache\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.142098 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.141889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kserve-provision-location\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.142098 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.142037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tmp-dir\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.142531 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.142325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-home\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.142531 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.142398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kserve-provision-location\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.142531 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.142490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-model-cache\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.144236 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.144208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-dshm\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.144629 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.144607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tls-certs\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.163626 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.163594 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9cz\" (UniqueName: \"kubernetes.io/projected/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kube-api-access-rw9cz\") pod \"tls-verification-test-kserve-55bf78c657-spd7n\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.253092 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.253004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:08.399382 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.399356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n"] Jul 28 18:49:08.401480 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:49:08.401453 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34b552c_7ab9_47cd_b4aa_05f0d385a3cc.slice/crio-0b83d11c4394845a49d8d785bd4a15815cbadf9437413e615c38dd0d7927acd0 WatchSource:0}: Error finding container 0b83d11c4394845a49d8d785bd4a15815cbadf9437413e615c38dd0d7927acd0: Status 404 returned error can't find the container with id 0b83d11c4394845a49d8d785bd4a15815cbadf9437413e615c38dd0d7927acd0 Jul 28 18:49:08.471439 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.471400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" event={"ID":"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc","Type":"ContainerStarted","Data":"89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21"} Jul 28 18:49:08.471627 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:08.471466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" event={"ID":"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc","Type":"ContainerStarted","Data":"0b83d11c4394845a49d8d785bd4a15815cbadf9437413e615c38dd0d7927acd0"} Jul 28 18:49:41.675584 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:41.675495 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"] Jul 28 18:49:41.685294 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:49:41.685255 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 18:49:41.685294 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:49:41.685294 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:49:41.685294 ip-10-0-131-29 kubenswrapper[2564]: > containerID="55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" cmd=["/bin/sleep","15"] Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:41.685311 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" message="" err=< Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:49:41.685324 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" containerName="main" containerID="cri-o://55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" Jul 28 18:49:41.685652 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:41.685362 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" containerName="main" containerID="cri-o://55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" gracePeriod=60 Jul 28 18:49:42.036409 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.036380 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:49:42.177284 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177248 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tmp-dir\") pod \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " Jul 28 18:49:42.177481 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177294 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-home\") pod \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " Jul 28 18:49:42.177481 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jthvz\" (UniqueName: \"kubernetes.io/projected/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-kube-api-access-jthvz\") pod \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " Jul 28 18:49:42.177481 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177378 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-dshm\") pod \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " Jul 28 18:49:42.177658 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177531 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tls-certs\") pod \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " Jul 28 18:49:42.177658 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177575 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e2bfc6e-a038-494f-a1f4-ea93137b70ea" (UID: "4e2bfc6e-a038-494f-a1f4-ea93137b70ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:42.177658 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177608 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-model-cache\") pod \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\" (UID: \"4e2bfc6e-a038-494f-a1f4-ea93137b70ea\") " Jul 28 18:49:42.177658 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177599 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-home" (OuterVolumeSpecName: "home") pod "4e2bfc6e-a038-494f-a1f4-ea93137b70ea" (UID: "4e2bfc6e-a038-494f-a1f4-ea93137b70ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:42.177967 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e2bfc6e-a038-494f-a1f4-ea93137b70ea" (UID: "4e2bfc6e-a038-494f-a1f4-ea93137b70ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:42.177967 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177964 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:49:42.178119 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.177981 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:49:42.179840 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.179814 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-dshm" (OuterVolumeSpecName: "dshm") pod "4e2bfc6e-a038-494f-a1f4-ea93137b70ea" (UID: "4e2bfc6e-a038-494f-a1f4-ea93137b70ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:42.179941 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.179894 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-kube-api-access-jthvz" (OuterVolumeSpecName: "kube-api-access-jthvz") pod "4e2bfc6e-a038-494f-a1f4-ea93137b70ea" (UID: "4e2bfc6e-a038-494f-a1f4-ea93137b70ea"). InnerVolumeSpecName "kube-api-access-jthvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:49:42.179992 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.179977 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e2bfc6e-a038-494f-a1f4-ea93137b70ea" (UID: "4e2bfc6e-a038-494f-a1f4-ea93137b70ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:49:42.279125 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.279027 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:49:42.279125 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.279060 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:49:42.279125 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.279069 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthvz\" (UniqueName: \"kubernetes.io/projected/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-kube-api-access-jthvz\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:49:42.279125 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.279077 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2bfc6e-a038-494f-a1f4-ea93137b70ea-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:49:42.641362 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.641324 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" containerID="55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" exitCode=0 Jul 28 18:49:42.641535 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.641375 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" Jul 28 18:49:42.641535 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.641400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4e2bfc6e-a038-494f-a1f4-ea93137b70ea","Type":"ContainerDied","Data":"55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac"} Jul 28 18:49:42.641535 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.641451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4e2bfc6e-a038-494f-a1f4-ea93137b70ea","Type":"ContainerDied","Data":"ef4ccd371666ea2d8c9dff51ffc735b587811d64e9cc05b0c091284c190fbc2f"} Jul 28 18:49:42.641535 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.641477 2564 scope.go:117] "RemoveContainer" containerID="55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" Jul 28 18:49:42.652749 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.652729 2564 scope.go:117] "RemoveContainer" containerID="55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" Jul 28 18:49:42.653095 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:49:42.653073 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac\": container with ID starting with 55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac not found: ID does not exist" containerID="55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac" Jul 28 18:49:42.653186 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.653102 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac"} err="failed to get container status \"55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac\": rpc error: code = NotFound desc = could not find container \"55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac\": container with ID starting with 55d3e38e2a94acd140c9c9bc3f80c121d97820722a56b4be9a2686b1f8309fac not found: ID does not exist" Jul 28 18:49:42.667081 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.667048 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"] Jul 28 18:49:42.671088 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:42.671059 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"] Jul 28 18:49:43.377724 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:43.377676 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" path="/var/lib/kubelet/pods/4e2bfc6e-a038-494f-a1f4-ea93137b70ea/volumes" Jul 28 18:49:49.679618 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:49.679582 2564 generic.go:356] "Generic (PLEG): container finished" podID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerID="89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21" exitCode=0 Jul 28 18:49:49.680038 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:49.679655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" event={"ID":"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc","Type":"ContainerDied","Data":"89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21"} Jul 28 18:49:50.686675 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:50.686642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" event={"ID":"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc","Type":"ContainerStarted","Data":"17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908"} Jul 28 18:49:50.712712 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:50.712647 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podStartSLOduration=2.680474695 podStartE2EDuration="43.712629326s" totalImagesPullingTime="0s" totalInitContainerRuntime="41.032154631s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:49:50.709860351 +0000 UTC m=+5223.862360575" watchObservedRunningTime="2026-07-28 18:49:50.712629326 +0000 UTC m=+5223.865129503" Jul 28 18:49:51.253195 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:51.253150 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:58.253857 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:58.253816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:49:58.255453 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:49:58.255422 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:50:08.253605 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:50:08.253554 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:50:18.254397 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:50:18.254350 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:50:28.254506 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:50:28.254461 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:50:38.254315 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:50:38.254267 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:50:48.253944 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:50:48.253900 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:50:58.253986 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:50:58.253932 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:51:08.254219 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:08.254163 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:51:18.254435 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:18.254325 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:51:28.254638 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:28.254574 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:51:38.254247 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:38.254190 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:51:48.254127 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:48.254073 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:51:58.264442 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:58.264403 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:51:58.272515 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:51:58.272484 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:52:13.058668 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:13.058626 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n"] Jul 28 18:52:28.081464 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.081419 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" containerID="cri-o://17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908" gracePeriod=45 Jul 28 18:52:28.265995 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.265952 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 28 18:52:28.841441 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.841414 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:52:28.958180 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tmp-dir\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958180 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-home\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958180 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958150 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-model-cache\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958180 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958168 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-dshm\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958537 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958218 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kserve-provision-location\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958537 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958242 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tls-certs\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958537 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958291 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw9cz\" (UniqueName: \"kubernetes.io/projected/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kube-api-access-rw9cz\") pod \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\" (UID: \"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc\") " Jul 28 18:52:28.958537 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958431 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:52:28.958796 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958555 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:28.958890 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.958865 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-home" (OuterVolumeSpecName: "home") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:52:28.960792 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.960730 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:52:28.960926 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.960862 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-dshm" (OuterVolumeSpecName: "dshm") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:52:28.961338 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.961303 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kube-api-access-rw9cz" (OuterVolumeSpecName: "kube-api-access-rw9cz") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "kube-api-access-rw9cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:52:28.977393 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:28.977315 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:52:29.021277 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.021237 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" (UID: "c34b552c-7ab9-47cd-b4aa-05f0d385a3cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:52:29.059518 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.059487 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:29.059518 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.059518 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:29.059518 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.059526 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:29.059707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.059535 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:29.059707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.059544 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:29.059707 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.059554 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw9cz\" (UniqueName: \"kubernetes.io/projected/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc-kube-api-access-rw9cz\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:52:29.540132 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.540092 2564 generic.go:356] "Generic (PLEG): container finished" podID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerID="17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908" exitCode=0 Jul 28 18:52:29.540586 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.540136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" event={"ID":"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc","Type":"ContainerDied","Data":"17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908"} Jul 28 18:52:29.540586 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.540203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" event={"ID":"c34b552c-7ab9-47cd-b4aa-05f0d385a3cc","Type":"ContainerDied","Data":"0b83d11c4394845a49d8d785bd4a15815cbadf9437413e615c38dd0d7927acd0"} Jul 28 18:52:29.540586 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.540226 2564 scope.go:117] "RemoveContainer" containerID="17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908" Jul 28 18:52:29.540586 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.540147 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n" Jul 28 18:52:29.549577 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.549561 2564 scope.go:117] "RemoveContainer" containerID="89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21" Jul 28 18:52:29.558164 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.558138 2564 scope.go:117] "RemoveContainer" containerID="17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908" Jul 28 18:52:29.558477 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:52:29.558449 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908\": container with ID starting with 17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908 not found: ID does not exist" containerID="17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908" Jul 28 18:52:29.558568 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.558487 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908"} err="failed to get container status \"17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908\": rpc error: code = NotFound desc = could not find container \"17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908\": container with ID starting with 17de9ead2bc1fef6d457ff4d677d792b92c0eae6c309bc9ec80de508e5081908 not found: ID does not exist" Jul 28 18:52:29.558568 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.558523 2564 scope.go:117] "RemoveContainer" containerID="89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21" Jul 28 18:52:29.558931 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:52:29.558910 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21\": container with ID starting with 89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21 not found: ID does not exist" containerID="89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21" Jul 28 18:52:29.559011 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.558935 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21"} err="failed to get container status \"89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21\": rpc error: code = NotFound desc = could not find container \"89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21\": container with ID starting with 89206a11add7b9e4d0671b3535b58db27f99c9679000fc8d6511861a14e96b21 not found: ID does not exist" Jul 28 18:52:29.561040 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.561016 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n"] Jul 28 18:52:29.565176 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:29.565151 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-spd7n"] Jul 28 18:52:31.384149 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:31.384109 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" path="/var/lib/kubelet/pods/c34b552c-7ab9-47cd-b4aa-05f0d385a3cc/volumes" Jul 28 18:52:47.924565 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:47.924435 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:52:47.942714 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:47.938720 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:52:51.697376 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697333 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6"] Jul 28 18:52:51.697793 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697724 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" containerName="main" Jul 28 18:52:51.697793 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697735 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" containerName="main" Jul 28 18:52:51.697793 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697746 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" Jul 28 18:52:51.697793 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697751 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" Jul 28 18:52:51.697939 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697807 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="storage-initializer" Jul 28 18:52:51.697939 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697814 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="storage-initializer" Jul 28 18:52:51.697939 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697896 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2bfc6e-a038-494f-a1f4-ea93137b70ea" containerName="main" Jul 28 18:52:51.697939 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.697907 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34b552c-7ab9-47cd-b4aa-05f0d385a3cc" containerName="main" Jul 28 18:52:51.702271 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.702249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.705254 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.705229 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4nwj4\"" Jul 28 18:52:51.705732 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.705419 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\"" Jul 28 18:52:51.705732 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.705573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\"" Jul 28 18:52:51.705732 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.705587 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\"" Jul 28 18:52:51.705732 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.705603 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\"" Jul 28 18:52:51.715084 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.715054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6"] Jul 28 18:52:51.749194 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.749194 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.749428 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.749428 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.749428 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjtg\" (UniqueName: \"kubernetes.io/projected/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kube-api-access-6bjtg\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.749428 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.749428 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.749379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850296 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850296 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850551 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjtg\" (UniqueName: \"kubernetes.io/projected/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kube-api-access-6bjtg\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850551 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850551 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850551 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850551 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850829 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850829 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850947 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.850947 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.850922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.852742 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.852716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.852930 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.852913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:51.859983 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:51.859959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjtg\" (UniqueName: \"kubernetes.io/projected/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kube-api-access-6bjtg\") pod \"prestop-hook-test-kserve-68f6bd7b79-2rpk6\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:52.013890 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:52.013785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:52:52.165410 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:52.165379 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6"] Jul 28 18:52:52.167125 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:52:52.167095 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb356d56a_0b83_489d_9b3c_bd9bb04a0625.slice/crio-4a6cf2bc376df1a245abf120608d3039ab8576744b1b3089a97178a1f64f4a54 WatchSource:0}: Error finding container 4a6cf2bc376df1a245abf120608d3039ab8576744b1b3089a97178a1f64f4a54: Status 404 returned error can't find the container with id 4a6cf2bc376df1a245abf120608d3039ab8576744b1b3089a97178a1f64f4a54 Jul 28 18:52:52.657671 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:52.657620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" event={"ID":"b356d56a-0b83-489d-9b3c-bd9bb04a0625","Type":"ContainerStarted","Data":"0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291"} Jul 28 18:52:52.657889 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:52:52.657700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" event={"ID":"b356d56a-0b83-489d-9b3c-bd9bb04a0625","Type":"ContainerStarted","Data":"4a6cf2bc376df1a245abf120608d3039ab8576744b1b3089a97178a1f64f4a54"} Jul 28 18:53:00.701099 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:00.701062 2564 generic.go:356] "Generic (PLEG): container finished" podID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerID="0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291" exitCode=0 Jul 28 18:53:00.701527 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:00.701110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" event={"ID":"b356d56a-0b83-489d-9b3c-bd9bb04a0625","Type":"ContainerDied","Data":"0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291"} Jul 28 18:53:01.709240 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:01.709202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" event={"ID":"b356d56a-0b83-489d-9b3c-bd9bb04a0625","Type":"ContainerStarted","Data":"641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba"} Jul 28 18:53:01.735818 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:01.735750 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podStartSLOduration=3.078728691 podStartE2EDuration="10.7357319s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.657003209s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:01.733644504 +0000 UTC m=+5414.886144699" watchObservedRunningTime="2026-07-28 18:53:01.7357319 +0000 UTC m=+5414.888232076" Jul 28 18:53:02.014870 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:02.014750 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:53:02.014870 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:02.014848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:53:02.016519 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:02.016465 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:53:12.014845 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:12.014799 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:53:22.015057 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:22.015010 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:53:32.014498 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:32.014446 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:53:42.014867 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:42.014810 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:53:52.015273 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:53:52.015226 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:54:02.014697 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:54:02.014642 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:54:12.014470 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:54:12.014383 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:54:22.015232 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:54:22.015176 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:54:32.014543 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:54:32.014498 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:54:42.015310 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:54:42.015244 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:54:52.014894 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:54:52.014845 2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused" Jul 28 18:55:02.024834 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:02.024796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:55:02.033008 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:02.032980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:55:06.378793 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:06.378740 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6"] Jul 28 18:55:21.401546 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:21.401484 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" containerID="cri-o://641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba" gracePeriod=45 Jul 28 18:55:22.077296 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.077267 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:55:22.107013 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.106977 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tls-certs\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107155 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107040 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-dshm\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107155 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kserve-provision-location\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107155 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-model-cache\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107155 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107126 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-home\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107155 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bjtg\" (UniqueName: \"kubernetes.io/projected/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kube-api-access-6bjtg\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107418 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107160 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tmp-dir\") pod \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\" (UID: \"b356d56a-0b83-489d-9b3c-bd9bb04a0625\") " Jul 28 18:55:22.107476 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107417 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-model-cache" (OuterVolumeSpecName: "model-cache") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:22.107568 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107538 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.107903 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.107872 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-home" (OuterVolumeSpecName: "home") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:22.109638 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.109592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-dshm" (OuterVolumeSpecName: "dshm") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:22.110074 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.109907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kube-api-access-6bjtg" (OuterVolumeSpecName: "kube-api-access-6bjtg") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "kube-api-access-6bjtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:55:22.110074 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.110033 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:55:22.121908 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.121845 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:22.192789 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.192667 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b356d56a-0b83-489d-9b3c-bd9bb04a0625" (UID: "b356d56a-0b83-489d-9b3c-bd9bb04a0625"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:22.208429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.208399 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.208429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.208428 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bjtg\" (UniqueName: \"kubernetes.io/projected/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kube-api-access-6bjtg\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.208633 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.208439 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.208633 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.208448 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b356d56a-0b83-489d-9b3c-bd9bb04a0625-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.208633 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.208456 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.208633 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.208464 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b356d56a-0b83-489d-9b3c-bd9bb04a0625-kserve-provision-location\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:55:22.462390 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.462305 2564 generic.go:356] "Generic (PLEG): container finished" podID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerID="641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba" exitCode=0 Jul 28 18:55:22.462390 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.462354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" event={"ID":"b356d56a-0b83-489d-9b3c-bd9bb04a0625","Type":"ContainerDied","Data":"641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba"} Jul 28 18:55:22.462390 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.462361 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" Jul 28 18:55:22.462963 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.462399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" event={"ID":"b356d56a-0b83-489d-9b3c-bd9bb04a0625","Type":"ContainerDied","Data":"4a6cf2bc376df1a245abf120608d3039ab8576744b1b3089a97178a1f64f4a54"} Jul 28 18:55:22.462963 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.462421 2564 scope.go:117] "RemoveContainer" containerID="641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba" Jul 28 18:55:22.472229 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.472207 2564 scope.go:117] "RemoveContainer" containerID="0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291" Jul 28 18:55:22.480777 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.480740 2564 scope.go:117] "RemoveContainer" containerID="641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba" Jul 28 18:55:22.481068 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:55:22.481049 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba\": container with ID starting with 641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba not found: ID does not exist" containerID="641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba" Jul 28 18:55:22.481134 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.481076 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba"} err="failed to get container status \"641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba\": rpc error: code = NotFound desc = could not find container \"641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba\": container with ID starting with 641713ad7c9165ec53a8775ca7ae16cece6763d2e287070a0118db08e8f77fba not found: ID does not exist" Jul 28 18:55:22.481134 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.481094 2564 scope.go:117] "RemoveContainer" containerID="0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291" Jul 28 18:55:22.481337 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:55:22.481316 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291\": container with ID starting with 0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291 not found: ID does not exist" containerID="0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291" Jul 28 18:55:22.481376 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.481344 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291"} err="failed to get container status \"0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291\": rpc error: code = NotFound desc = could not find container \"0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291\": container with ID starting with 0be7a14fce7c073463260cf6a8d5fcc84296352402316afca0480537cd496291 not found: ID does not exist" Jul 28 18:55:22.491675 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.491646 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6"] Jul 28 18:55:22.496744 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:22.496715 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6"] Jul 28 18:55:23.025710 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:23.025644 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-2rpk6" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 28 18:55:23.381157 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:55:23.381120 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" path="/var/lib/kubelet/pods/b356d56a-0b83-489d-9b3c-bd9bb04a0625/volumes" Jul 28 18:56:39.184577 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.184538 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8"] Jul 28 18:56:39.185139 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.184975 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" Jul 28 18:56:39.185139 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.184987 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" Jul 28 18:56:39.185139 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.185014 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="storage-initializer" Jul 28 18:56:39.185139 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.185019 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="storage-initializer" Jul 28 18:56:39.185139 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.185082 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b356d56a-0b83-489d-9b3c-bd9bb04a0625" containerName="main" Jul 28 18:56:39.188238 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.188219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.196871 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.196843 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\"" Jul 28 18:56:39.198312 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.198290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-hsm69\"" Jul 28 18:56:39.198393 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.198319 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\"" Jul 28 18:56:39.198393 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.198290 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\"" Jul 28 18:56:39.207670 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.207640 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8"] Jul 28 18:56:39.310647 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.310611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.310647 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.310648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859e135a-2d9b-4d9e-94ca-d06631a4761f-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.310952 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.310674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.310952 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.310726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjtb\" (UniqueName: \"kubernetes.io/projected/859e135a-2d9b-4d9e-94ca-d06631a4761f-kube-api-access-6xjtb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.310952 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.310881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.310952 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.310904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.411543 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.411503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.411543 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.411544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.411853 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.411592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.411853 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.411613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859e135a-2d9b-4d9e-94ca-d06631a4761f-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.411853 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.411630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.411853 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.411651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjtb\" (UniqueName: \"kubernetes.io/projected/859e135a-2d9b-4d9e-94ca-d06631a4761f-kube-api-access-6xjtb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.412071 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.412016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.412137 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.412095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.412137 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.412102 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.414083 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.414062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.414322 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.414302 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859e135a-2d9b-4d9e-94ca-d06631a4761f-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.425306 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.425277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjtb\" (UniqueName: \"kubernetes.io/projected/859e135a-2d9b-4d9e-94ca-d06631a4761f-kube-api-access-6xjtb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qq2n8\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.499609 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.499512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:39.650569 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.650544 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8"] Jul 28 18:56:39.653119 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:56:39.653089 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859e135a_2d9b_4d9e_94ca_d06631a4761f.slice/crio-d5a967f2de6a43f8c38d0ff336dd6a6a9dce1453564e68adc4cda7815f01545f WatchSource:0}: Error finding container d5a967f2de6a43f8c38d0ff336dd6a6a9dce1453564e68adc4cda7815f01545f: Status 404 returned error can't find the container with id d5a967f2de6a43f8c38d0ff336dd6a6a9dce1453564e68adc4cda7815f01545f Jul 28 18:56:39.853028 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.852979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" event={"ID":"859e135a-2d9b-4d9e-94ca-d06631a4761f","Type":"ContainerStarted","Data":"a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c"} Jul 28 18:56:39.853231 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.853048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" event={"ID":"859e135a-2d9b-4d9e-94ca-d06631a4761f","Type":"ContainerStarted","Data":"d5a967f2de6a43f8c38d0ff336dd6a6a9dce1453564e68adc4cda7815f01545f"} Jul 28 18:56:39.883544 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:39.883484 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" podStartSLOduration=0.883467118 podStartE2EDuration="883.467118ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:39.88100817 +0000 UTC m=+5633.033508349" watchObservedRunningTime="2026-07-28 18:56:39.883467118 +0000 UTC m=+5633.035967294" Jul 28 18:56:40.500217 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:40.500178 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:49.499782 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:49.499669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:49.512120 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:49.512090 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:56:49.918787 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:56:49.918731 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:57:47.961662 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:47.961546 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:57:47.975916 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:47.975888 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 18:57:49.158606 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.158570 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8"] Jul 28 18:57:49.168290 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:57:49.168251 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 28 18:57:49.168290 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:57:49.168290 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:57:49.168290 ip-10-0-131-29 kubenswrapper[2564]: > containerID="a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" cmd=["/bin/sleep","15"] Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.168307 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" message="" err=< Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: > Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:57:49.168319 2564 kuberuntime_container.go:755] "PreStop hook failed" err=< Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" podUID="859e135a-2d9b-4d9e-94ca-d06631a4761f" containerName="main" containerID="cri-o://a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" Jul 28 18:57:49.168550 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.168367 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" podUID="859e135a-2d9b-4d9e-94ca-d06631a4761f" containerName="main" containerID="cri-o://a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" gracePeriod=60 Jul 28 18:57:49.523637 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.523610 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:57:49.568138 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjtb\" (UniqueName: \"kubernetes.io/projected/859e135a-2d9b-4d9e-94ca-d06631a4761f-kube-api-access-6xjtb\") pod \"859e135a-2d9b-4d9e-94ca-d06631a4761f\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " Jul 28 18:57:49.568325 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568153 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-home\") pod \"859e135a-2d9b-4d9e-94ca-d06631a4761f\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " Jul 28 18:57:49.568325 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-dshm\") pod \"859e135a-2d9b-4d9e-94ca-d06631a4761f\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " Jul 28 18:57:49.568325 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568236 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859e135a-2d9b-4d9e-94ca-d06631a4761f-tls-certs\") pod \"859e135a-2d9b-4d9e-94ca-d06631a4761f\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " Jul 28 18:57:49.568325 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-tmp-dir\") pod \"859e135a-2d9b-4d9e-94ca-d06631a4761f\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " Jul 28 18:57:49.568522 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568353 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-model-cache\") pod \"859e135a-2d9b-4d9e-94ca-d06631a4761f\" (UID: \"859e135a-2d9b-4d9e-94ca-d06631a4761f\") " Jul 28 18:57:49.568580 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568498 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-home" (OuterVolumeSpecName: "home") pod "859e135a-2d9b-4d9e-94ca-d06631a4761f" (UID: "859e135a-2d9b-4d9e-94ca-d06631a4761f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:57:49.568636 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568569 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "859e135a-2d9b-4d9e-94ca-d06631a4761f" (UID: "859e135a-2d9b-4d9e-94ca-d06631a4761f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:57:49.568726 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568697 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-model-cache" (OuterVolumeSpecName: "model-cache") pod "859e135a-2d9b-4d9e-94ca-d06631a4761f" (UID: "859e135a-2d9b-4d9e-94ca-d06631a4761f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:57:49.568726 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568726 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-home\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:57:49.568938 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.568740 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-tmp-dir\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:57:49.570527 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.570501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859e135a-2d9b-4d9e-94ca-d06631a4761f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "859e135a-2d9b-4d9e-94ca-d06631a4761f" (UID: "859e135a-2d9b-4d9e-94ca-d06631a4761f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:57:49.570662 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.570502 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-dshm" (OuterVolumeSpecName: "dshm") pod "859e135a-2d9b-4d9e-94ca-d06631a4761f" (UID: "859e135a-2d9b-4d9e-94ca-d06631a4761f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:57:49.570662 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.570619 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859e135a-2d9b-4d9e-94ca-d06631a4761f-kube-api-access-6xjtb" (OuterVolumeSpecName: "kube-api-access-6xjtb") pod "859e135a-2d9b-4d9e-94ca-d06631a4761f" (UID: "859e135a-2d9b-4d9e-94ca-d06631a4761f"). InnerVolumeSpecName "kube-api-access-6xjtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:57:49.669977 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.669891 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-model-cache\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:57:49.669977 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.669919 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjtb\" (UniqueName: \"kubernetes.io/projected/859e135a-2d9b-4d9e-94ca-d06631a4761f-kube-api-access-6xjtb\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:57:49.669977 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.669929 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859e135a-2d9b-4d9e-94ca-d06631a4761f-dshm\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:57:49.669977 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:49.669938 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859e135a-2d9b-4d9e-94ca-d06631a4761f-tls-certs\") on node \"ip-10-0-131-29.ec2.internal\" DevicePath \"\"" Jul 28 18:57:50.219157 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.219126 2564 generic.go:356] "Generic (PLEG): container finished" podID="859e135a-2d9b-4d9e-94ca-d06631a4761f" containerID="a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" exitCode=0 Jul 28 18:57:50.219666 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.219175 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" Jul 28 18:57:50.219666 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.219217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" event={"ID":"859e135a-2d9b-4d9e-94ca-d06631a4761f","Type":"ContainerDied","Data":"a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c"} Jul 28 18:57:50.219666 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.219277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8" event={"ID":"859e135a-2d9b-4d9e-94ca-d06631a4761f","Type":"ContainerDied","Data":"d5a967f2de6a43f8c38d0ff336dd6a6a9dce1453564e68adc4cda7815f01545f"} Jul 28 18:57:50.219666 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.219309 2564 scope.go:117] "RemoveContainer" containerID="a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" Jul 28 18:57:50.229135 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.229116 2564 scope.go:117] "RemoveContainer" containerID="a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" Jul 28 18:57:50.229436 ip-10-0-131-29 kubenswrapper[2564]: E0728 18:57:50.229419 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c\": container with ID starting with a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c not found: ID does not exist" containerID="a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c" Jul 28 18:57:50.229500 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.229447 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c"} err="failed to get container status \"a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c\": rpc error: code = NotFound desc = could not find container \"a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c\": container with ID starting with a06d9829a633d1e7fdda486d1a3974c6f52edb30706646db4dea63b7becbec3c not found: ID does not exist" Jul 28 18:57:50.242315 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.242278 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8"] Jul 28 18:57:50.247298 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:50.247266 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qq2n8"] Jul 28 18:57:51.377466 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:57:51.377426 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859e135a-2d9b-4d9e-94ca-d06631a4761f" path="/var/lib/kubelet/pods/859e135a-2d9b-4d9e-94ca-d06631a4761f/volumes" Jul 28 18:58:23.466899 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.466867 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-779d8dc47c-bkr24"] Jul 28 18:58:23.467424 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.467262 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859e135a-2d9b-4d9e-94ca-d06631a4761f" containerName="main" Jul 28 18:58:23.467424 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.467278 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="859e135a-2d9b-4d9e-94ca-d06631a4761f" containerName="main" Jul 28 18:58:23.467424 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.467354 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="859e135a-2d9b-4d9e-94ca-d06631a4761f" containerName="main" Jul 28 18:58:23.470625 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.470605 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.475398 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.475365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 18:58:23.475540 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.475365 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 28 18:58:23.475540 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.475365 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-n544f\"" Jul 28 18:58:23.475540 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.475441 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 18:58:23.480141 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.480112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-779d8dc47c-bkr24"] Jul 28 18:58:23.482956 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.482933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c986c7c-6715-4232-ba09-96ce153c705d-cert\") pod \"llmisvc-controller-manager-779d8dc47c-bkr24\" (UID: \"1c986c7c-6715-4232-ba09-96ce153c705d\") " pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.483060 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.482979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffqd\" (UniqueName: \"kubernetes.io/projected/1c986c7c-6715-4232-ba09-96ce153c705d-kube-api-access-vffqd\") pod \"llmisvc-controller-manager-779d8dc47c-bkr24\" (UID: \"1c986c7c-6715-4232-ba09-96ce153c705d\") " pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.584178 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.584132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c986c7c-6715-4232-ba09-96ce153c705d-cert\") pod \"llmisvc-controller-manager-779d8dc47c-bkr24\" (UID: \"1c986c7c-6715-4232-ba09-96ce153c705d\") " pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.584374 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.584200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffqd\" (UniqueName: \"kubernetes.io/projected/1c986c7c-6715-4232-ba09-96ce153c705d-kube-api-access-vffqd\") pod \"llmisvc-controller-manager-779d8dc47c-bkr24\" (UID: \"1c986c7c-6715-4232-ba09-96ce153c705d\") " pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.586690 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.586660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c986c7c-6715-4232-ba09-96ce153c705d-cert\") pod \"llmisvc-controller-manager-779d8dc47c-bkr24\" (UID: \"1c986c7c-6715-4232-ba09-96ce153c705d\") " pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.593429 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.593400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffqd\" (UniqueName: \"kubernetes.io/projected/1c986c7c-6715-4232-ba09-96ce153c705d-kube-api-access-vffqd\") pod \"llmisvc-controller-manager-779d8dc47c-bkr24\" (UID: \"1c986c7c-6715-4232-ba09-96ce153c705d\") " pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.781584 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.781489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:23.921097 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.921068 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-779d8dc47c-bkr24"] Jul 28 18:58:23.923585 ip-10-0-131-29 kubenswrapper[2564]: W0728 18:58:23.923556 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c986c7c_6715_4232_ba09_96ce153c705d.slice/crio-da982ac31de0c2c147a750a9fcac56559d5f751ca69cd6490d84912dcef0d943 WatchSource:0}: Error finding container da982ac31de0c2c147a750a9fcac56559d5f751ca69cd6490d84912dcef0d943: Status 404 returned error can't find the container with id da982ac31de0c2c147a750a9fcac56559d5f751ca69cd6490d84912dcef0d943 Jul 28 18:58:23.924960 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:23.924939 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:58:24.384459 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:24.384421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" event={"ID":"1c986c7c-6715-4232-ba09-96ce153c705d","Type":"ContainerStarted","Data":"da982ac31de0c2c147a750a9fcac56559d5f751ca69cd6490d84912dcef0d943"} Jul 28 18:58:27.401885 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:27.401838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" event={"ID":"1c986c7c-6715-4232-ba09-96ce153c705d","Type":"ContainerStarted","Data":"95169c742271917c4ac7e4425b7e9943b15c48c5e27a907e299cccaf4ca28c64"} Jul 28 18:58:27.402289 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:27.401911 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 18:58:27.423294 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:27.423238 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" podStartSLOduration=1.659967593 podStartE2EDuration="4.423221683s" totalImagesPullingTime="2.76325409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:58:27.420584488 +0000 UTC m=+5740.573084715" watchObservedRunningTime="2026-07-28 18:58:27.423221683 +0000 UTC m=+5740.575721860" Jul 28 18:58:58.409316 ip-10-0-131-29 kubenswrapper[2564]: I0728 18:58:58.409284 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-779d8dc47c-bkr24" Jul 28 19:02:47.995880 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:02:47.995750 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 19:02:48.008974 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:02:48.008951 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 19:06:02.599655 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:02.599622 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:02.627013 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:02.626986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:02.769486 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:02.769429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:02.779215 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:02.779191 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:03.953560 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:03.953531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:03.975725 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:03.975695 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:04.140010 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:04.139973 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:04.219952 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:04.219872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:05.287178 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:05.287128 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:05.303065 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:05.303041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:05.445844 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:05.445815 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:05.459989 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:05.459958 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:06.475331 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:06.475271 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:06.490043 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:06.490019 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:06.633277 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:06.633240 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:06.647930 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:06.647899 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:07.661998 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:07.661966 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:07.678833 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:07.678806 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:07.823325 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:07.823289 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:07.837667 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:07.837639 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:08.847519 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:08.847478 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:08.862500 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:08.862473 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:09.004419 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:09.004379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:09.018070 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:09.018041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:10.058841 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:10.058767 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:10.072606 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:10.072577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:10.214010 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:10.213981 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:10.227546 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:10.227520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:11.232657 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:11.232632 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:11.246079 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:11.246052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:11.393676 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:11.393645 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:11.406499 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:11.406469 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:12.433437 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:12.433406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:12.448049 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:12.448025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:12.590988 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:12.590946 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:12.604132 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:12.604100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:13.646586 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:13.646554 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:13.660211 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:13.660182 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:13.816714 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:13.816681 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:13.828722 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:13.828691 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:14.853352 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:14.853325 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:14.870558 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:14.870532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:14.997750 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:14.997714 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:15.037302 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:15.037272 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:16.093379 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:16.093352 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:16.106020 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:16.105994 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:16.245704 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:16.245669 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:16.258211 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:16.258184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:17.190298 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:17.190245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:17.205479 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:17.205451 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:17.358296 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:17.358258 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:17.370724 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:17.370696 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:18.399717 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:18.399690 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:18.413790 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:18.413750 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:18.558365 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:18.558333 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:18.572709 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:18.572678 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:19.617277 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:19.617246 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:19.631473 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:19.631447 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:19.771320 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:19.771284 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:19.784046 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:19.784014 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:20.805840 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:20.805791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:20.825438 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:20.825413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:20.965670 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:20.965640 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:20.982911 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:20.982882 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:22.129517 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:22.129488 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:22.147103 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:22.147072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:22.286946 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:22.286913 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:22.300901 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:22.300872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:23.319664 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:23.319635 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:23.333542 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:23.333518 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:23.466909 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:23.466875 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:23.482950 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:23.482921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:24.504492 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:24.504461 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:24.518446 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:24.518422 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:24.656192 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:24.656155 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:24.668800 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:24.668747 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:25.691566 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:25.691535 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:25.706501 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:25.706469 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:25.851905 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:25.851865 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:25.865164 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:25.865133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:26.889670 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:26.889642 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:26.905091 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:26.905062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:27.045790 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:27.045744 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:27.058487 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:27.058460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:28.083539 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:28.083511 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:28.098092 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:28.098065 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:28.248846 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:28.248812 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:28.262053 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:28.262024 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:29.322213 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:29.322188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:29.338405 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:29.338380 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:29.483437 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:29.483401 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:29.498670 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:29.498642 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:30.536493 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:30.536464 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:30.569182 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:30.569152 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:30.716851 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:30.716818 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:30.731071 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:30.731045 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:31.842812 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:31.842749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:31.857264 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:31.857235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:31.999098 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:31.999062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:32.012581 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:32.012557 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:34.276622 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.276585 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd9v9/must-gather-tn7p8"] Jul 28 19:06:34.280146 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.280121 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.283046 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.283017 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd9v9\"/\"kube-root-ca.crt\"" Jul 28 19:06:34.284307 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.284288 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd9v9\"/\"openshift-service-ca.crt\"" Jul 28 19:06:34.284419 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.284318 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd9v9\"/\"default-dockercfg-k9wgm\"" Jul 28 19:06:34.290547 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.290515 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/must-gather-tn7p8"] Jul 28 19:06:34.406203 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.406155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8f43197-7d0f-4ca3-b6f6-b3da32579150-must-gather-output\") pod \"must-gather-tn7p8\" (UID: \"c8f43197-7d0f-4ca3-b6f6-b3da32579150\") " pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.406391 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.406284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzfp\" (UniqueName: \"kubernetes.io/projected/c8f43197-7d0f-4ca3-b6f6-b3da32579150-kube-api-access-gdzfp\") pod \"must-gather-tn7p8\" (UID: \"c8f43197-7d0f-4ca3-b6f6-b3da32579150\") " pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.507583 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.507537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8f43197-7d0f-4ca3-b6f6-b3da32579150-must-gather-output\") pod \"must-gather-tn7p8\" (UID: \"c8f43197-7d0f-4ca3-b6f6-b3da32579150\") " pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.507814 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.507637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzfp\" (UniqueName: \"kubernetes.io/projected/c8f43197-7d0f-4ca3-b6f6-b3da32579150-kube-api-access-gdzfp\") pod \"must-gather-tn7p8\" (UID: \"c8f43197-7d0f-4ca3-b6f6-b3da32579150\") " pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.507952 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.507929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8f43197-7d0f-4ca3-b6f6-b3da32579150-must-gather-output\") pod \"must-gather-tn7p8\" (UID: \"c8f43197-7d0f-4ca3-b6f6-b3da32579150\") " pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.515917 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.515886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzfp\" (UniqueName: \"kubernetes.io/projected/c8f43197-7d0f-4ca3-b6f6-b3da32579150-kube-api-access-gdzfp\") pod \"must-gather-tn7p8\" (UID: \"c8f43197-7d0f-4ca3-b6f6-b3da32579150\") " pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.591005 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.590966 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/must-gather-tn7p8" Jul 28 19:06:34.735746 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.735713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/must-gather-tn7p8"] Jul 28 19:06:34.738481 ip-10-0-131-29 kubenswrapper[2564]: W0728 19:06:34.738451 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f43197_7d0f_4ca3_b6f6_b3da32579150.slice/crio-4ea780877934f3b9a70ea9a12389f13c1ab78d1d543ed66f405b281ba65c3974 WatchSource:0}: Error finding container 4ea780877934f3b9a70ea9a12389f13c1ab78d1d543ed66f405b281ba65c3974: Status 404 returned error can't find the container with id 4ea780877934f3b9a70ea9a12389f13c1ab78d1d543ed66f405b281ba65c3974 Jul 28 19:06:34.740395 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.740377 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 19:06:34.875164 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:34.875067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/must-gather-tn7p8" event={"ID":"c8f43197-7d0f-4ca3-b6f6-b3da32579150","Type":"ContainerStarted","Data":"4ea780877934f3b9a70ea9a12389f13c1ab78d1d543ed66f405b281ba65c3974"} Jul 28 19:06:35.884664 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:35.884628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/must-gather-tn7p8" event={"ID":"c8f43197-7d0f-4ca3-b6f6-b3da32579150","Type":"ContainerStarted","Data":"40003867539a82d5a4ac80a58e4341d5b0d27b1d8a40799d2717b0e4fbafd1c1"} Jul 28 19:06:35.885160 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:35.884677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/must-gather-tn7p8" event={"ID":"c8f43197-7d0f-4ca3-b6f6-b3da32579150","Type":"ContainerStarted","Data":"f70f19cc1692dcf6cab0dd3e2728f406c04003860efe3ba76a4436bb21c32394"} Jul 28 19:06:37.244548 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:37.244486 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vv2ch_0bfc7e4f-dad6-476a-9246-ac413de3d950/global-pull-secret-syncer/0.log" Jul 28 19:06:37.499783 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:37.499674 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g5cl4_0c9a5cf5-5a90-44c8-a3de-ac7b4d24b99d/konnectivity-agent/0.log" Jul 28 19:06:37.536213 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:37.536176 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-29.ec2.internal_85cc45522d7d542e0d84295a649d3342/haproxy/0.log" Jul 28 19:06:41.936977 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:41.936941 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-nkbjm_d1b822ff-8d7b-4510-aa67-0ff38770843b/manager/0.log" Jul 28 19:06:41.989140 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:41.989103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-99wjj_492fed35-d0ad-40b6-af96-fedd3e3c4b64/kuadrant-console-plugin/0.log" Jul 28 19:06:42.195809 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:42.195312 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-4zxvt_42608b8a-a47a-4ce6-a66b-48f09e393ff9/manager/0.log" Jul 28 19:06:42.236586 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:42.236556 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zllks_6893502a-4823-4003-91d2-70c3fa5dedeb/limitador/0.log" Jul 28 19:06:44.095547 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.095450 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-9mw5h_023e50f8-ef13-4635-b097-ef800db19746/cluster-monitoring-operator/0.log" Jul 28 19:06:44.130779 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.130075 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-kxrdb_3ff56a99-8ba5-4a79-ba71-1a7e393d64f4/kube-state-metrics/0.log" Jul 28 19:06:44.163999 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.163900 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-kxrdb_3ff56a99-8ba5-4a79-ba71-1a7e393d64f4/kube-rbac-proxy-main/0.log" Jul 28 19:06:44.193625 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.193584 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-kxrdb_3ff56a99-8ba5-4a79-ba71-1a7e393d64f4/kube-rbac-proxy-self/0.log" Jul 28 19:06:44.258964 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.258924 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-m9dql_ceff7ea7-5a00-4ed1-bbc6-94633cefaaa7/monitoring-plugin/0.log" Jul 28 19:06:44.393911 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.393824 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmtz_7006fefc-7d0d-4b0a-a026-83a18e65014b/node-exporter/0.log" Jul 28 19:06:44.418421 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.418386 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmtz_7006fefc-7d0d-4b0a-a026-83a18e65014b/kube-rbac-proxy/0.log" Jul 28 19:06:44.442700 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.442669 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmtz_7006fefc-7d0d-4b0a-a026-83a18e65014b/init-textfile/0.log" Jul 28 19:06:44.558779 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.558731 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nwgmw_5dce10a2-a57e-4da6-a1da-e98730fa9897/kube-rbac-proxy-main/0.log" Jul 28 19:06:44.591881 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.591834 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nwgmw_5dce10a2-a57e-4da6-a1da-e98730fa9897/kube-rbac-proxy-self/0.log" Jul 28 19:06:44.626551 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.626521 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nwgmw_5dce10a2-a57e-4da6-a1da-e98730fa9897/openshift-state-metrics/0.log" Jul 28 19:06:44.680443 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.680353 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/prometheus/0.log" Jul 28 19:06:44.702579 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.702545 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/config-reloader/0.log" Jul 28 19:06:44.726467 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.726435 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/thanos-sidecar/0.log" Jul 28 19:06:44.766740 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.766705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/kube-rbac-proxy-web/0.log" Jul 28 19:06:44.796920 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.796887 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/kube-rbac-proxy/0.log" Jul 28 19:06:44.825537 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.825504 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/kube-rbac-proxy-thanos/0.log" Jul 28 19:06:44.851441 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.851413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_425ffd7c-0cd3-447d-91b6-1605ace2af7a/init-config-reloader/0.log" Jul 28 19:06:44.889049 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.889015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-f2z88_51fd1b3f-0da5-4635-aff9-775933b2f13b/prometheus-operator/0.log" Jul 28 19:06:44.912897 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:44.912868 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-f2z88_51fd1b3f-0da5-4635-aff9-775933b2f13b/kube-rbac-proxy/0.log" Jul 28 19:06:45.092820 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:45.092718 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cdb8d69-rlllz_0fefb1e4-4bbc-482c-9b23-40390e1a66c1/thanos-query/0.log" Jul 28 19:06:45.117201 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:45.117171 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cdb8d69-rlllz_0fefb1e4-4bbc-482c-9b23-40390e1a66c1/kube-rbac-proxy-web/0.log" Jul 28 19:06:45.140807 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:45.140775 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cdb8d69-rlllz_0fefb1e4-4bbc-482c-9b23-40390e1a66c1/kube-rbac-proxy/0.log" Jul 28 19:06:45.165665 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:45.165636 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cdb8d69-rlllz_0fefb1e4-4bbc-482c-9b23-40390e1a66c1/prom-label-proxy/0.log" Jul 28 19:06:45.194408 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:45.194327 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cdb8d69-rlllz_0fefb1e4-4bbc-482c-9b23-40390e1a66c1/kube-rbac-proxy-rules/0.log" Jul 28 19:06:45.220774 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:45.220727 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cdb8d69-rlllz_0fefb1e4-4bbc-482c-9b23-40390e1a66c1/kube-rbac-proxy-metrics/0.log" Jul 28 19:06:46.660079 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:46.660050 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-zb6h5_f135a084-91da-49cb-8a7e-60b9601ab0b4/networking-console-plugin/0.log" Jul 28 19:06:47.242828 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:47.242799 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/1.log" Jul 28 19:06:47.248459 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:47.248426 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rrhzz_e25edec6-93b9-4ef7-95a4-31fb4ffcbf22/console-operator/2.log" Jul 28 19:06:47.778713 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:47.778684 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76f56bf77c-cn6qr_7ee7af80-c0c4-4bc4-b2ef-b9ca29c09646/console/0.log" Jul 28 19:06:47.829887 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:47.829858 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-m2kzp_4783cdd8-1ae0-4520-88b2-c5fc3faf4159/download-server/0.log" Jul 28 19:06:48.212980 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.212915 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rd9v9/must-gather-tn7p8" podStartSLOduration=13.434746021 podStartE2EDuration="14.212893537s" totalImagesPullingTime="778.147516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:06:35.903616502 +0000 UTC m=+6229.056116681" watchObservedRunningTime="2026-07-28 19:06:48.212893537 +0000 UTC m=+6241.365393717" Jul 28 19:06:48.215684 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.215648 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x"] Jul 28 19:06:48.224014 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.223973 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.228575 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.228539 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x"] Jul 28 19:06:48.368307 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.368264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-sys\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.368630 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.368610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-lib-modules\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.368807 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.368785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6s6\" (UniqueName: \"kubernetes.io/projected/b2cbfbd8-9333-4f9b-9507-652d9831b05f-kube-api-access-df6s6\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.368972 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.368958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-podres\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.369184 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.369165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-proc\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470357 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-proc\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470357 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-sys\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470357 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-lib-modules\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470602 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6s6\" (UniqueName: \"kubernetes.io/projected/b2cbfbd8-9333-4f9b-9507-652d9831b05f-kube-api-access-df6s6\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470602 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-sys\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470602 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-podres\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470602 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-proc\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470602 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-podres\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.470602 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.470523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2cbfbd8-9333-4f9b-9507-652d9831b05f-lib-modules\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.479404 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.479370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6s6\" (UniqueName: \"kubernetes.io/projected/b2cbfbd8-9333-4f9b-9507-652d9831b05f-kube-api-access-df6s6\") pod \"perf-node-gather-daemonset-pks2x\" (UID: \"b2cbfbd8-9333-4f9b-9507-652d9831b05f\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.495925 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.495896 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-klwbj_b57768de-3577-447d-9cf1-85dc72dc600a/volume-data-source-validator/0.log" Jul 28 19:06:48.544738 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.544699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:48.726704 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.726599 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x"] Jul 28 19:06:48.731222 ip-10-0-131-29 kubenswrapper[2564]: W0728 19:06:48.731183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb2cbfbd8_9333_4f9b_9507_652d9831b05f.slice/crio-90dd80fa5276db992ea98847c55580fcfb5a7bf3edc204f0842914c033a54e2f WatchSource:0}: Error finding container 90dd80fa5276db992ea98847c55580fcfb5a7bf3edc204f0842914c033a54e2f: Status 404 returned error can't find the container with id 90dd80fa5276db992ea98847c55580fcfb5a7bf3edc204f0842914c033a54e2f Jul 28 19:06:48.988142 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.988054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" event={"ID":"b2cbfbd8-9333-4f9b-9507-652d9831b05f","Type":"ContainerStarted","Data":"ec8413fd594e6f0129552866068ce09480f285672d401bcf53c5142f65d5fa0f"} Jul 28 19:06:48.988142 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.988121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" event={"ID":"b2cbfbd8-9333-4f9b-9507-652d9831b05f","Type":"ContainerStarted","Data":"90dd80fa5276db992ea98847c55580fcfb5a7bf3edc204f0842914c033a54e2f"} Jul 28 19:06:48.988565 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:48.988155 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:49.009736 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:49.009661 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" podStartSLOduration=1.009636462 podStartE2EDuration="1.009636462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:06:49.007014857 +0000 UTC m=+6242.159515035" watchObservedRunningTime="2026-07-28 19:06:49.009636462 +0000 UTC m=+6242.162136639" Jul 28 19:06:49.571151 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:49.571122 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25cpf_4507d88d-12f1-40c0-8dc1-7d16bb80be4d/dns/0.log" Jul 28 19:06:49.593661 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:49.593633 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25cpf_4507d88d-12f1-40c0-8dc1-7d16bb80be4d/kube-rbac-proxy/0.log" Jul 28 19:06:49.741517 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:49.741490 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9ssz_01084945-8e46-41a8-a4eb-75d6ab25b232/dns-node-resolver/0.log" Jul 28 19:06:50.453471 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:50.453438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fl6jf_f1206b64-5e1d-4b68-b012-a4963d2d9ccf/node-ca/0.log" Jul 28 19:06:52.533613 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:52.533579 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t22pn_0264261c-056b-41f5-a232-c1905715ec4c/serve-healthcheck-canary/0.log" Jul 28 19:06:53.016447 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:53.016410 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-msvxc_5b0bb629-7283-4f68-8e90-c392c834a0e5/insights-operator/0.log" Jul 28 19:06:53.019691 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:53.019664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-msvxc_5b0bb629-7283-4f68-8e90-c392c834a0e5/insights-operator/1.log" Jul 28 19:06:53.183022 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:53.182993 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shhb8_c428648b-bbbc-434c-9d53-2629cdbd4935/kube-rbac-proxy/0.log" Jul 28 19:06:53.206538 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:53.206505 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shhb8_c428648b-bbbc-434c-9d53-2629cdbd4935/exporter/0.log" Jul 28 19:06:53.230990 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:53.230960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shhb8_c428648b-bbbc-434c-9d53-2629cdbd4935/extractor/0.log" Jul 28 19:06:55.009549 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:55.009518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-pks2x" Jul 28 19:06:56.287158 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:56.287130 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-fd6565d47-n59f7_b84f41a6-b6ee-41ce-86bc-27fabb9352c6/manager/0.log" Jul 28 19:06:57.143482 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:06:57.143430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-779d8dc47c-bkr24_1c986c7c-6715-4232-ba09-96ce153c705d/manager/0.log" Jul 28 19:07:05.157339 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.157308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-72bsm_2ef12920-6002-4d48-8959-4dcc9ad1d7d1/kube-multus/0.log" Jul 28 19:07:05.633125 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.633098 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/kube-multus-additional-cni-plugins/0.log" Jul 28 19:07:05.658103 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.658075 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/egress-router-binary-copy/0.log" Jul 28 19:07:05.686793 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.681834 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/cni-plugins/0.log" Jul 28 19:07:05.712910 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.712874 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/bond-cni-plugin/0.log" Jul 28 19:07:05.736498 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.736462 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/routeoverride-cni/0.log" Jul 28 19:07:05.761253 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.761222 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/whereabouts-cni-bincopy/0.log" Jul 28 19:07:05.785899 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.785871 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lr2pj_2b2becf2-702b-4344-93bb-f18eab272dd5/whereabouts-cni/0.log" Jul 28 19:07:05.952547 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.952520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n8rn6_669bb6ec-59b7-47cc-b345-43db24f5c868/network-metrics-daemon/0.log" Jul 28 19:07:05.972821 ip-10-0-131-29 kubenswrapper[2564]: I0728 19:07:05.972786 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n8rn6_669bb6ec-59b7-47cc-b345-43db24f5c868/kube-rbac-proxy/0.log"