Sep 16 18:45:36.782634 ip-10-0-128-157 systemd[1]: Starting Kubernetes Kubelet... Sep 16 18:45:37.164291 ip-10-0-128-157 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:37.164291 ip-10-0-128-157 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 18:45:37.164291 ip-10-0-128-157 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:37.164291 ip-10-0-128-157 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 18:45:37.164291 ip-10-0-128-157 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:37.165373 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.165298 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 18:45:37.169499 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169484 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:37.169499 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169498 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169502 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169505 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169508 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169511 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169514 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169516 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169519 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169521 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169524 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169526 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169529 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169531 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169534 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169536 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169539 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169541 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169544 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169550 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169552 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:37.169560 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169555 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169558 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169561 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169564 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169567 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169570 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169573 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169575 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169578 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169581 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169583 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169585 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169588 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169590 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169594 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169598 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169601 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169604 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169606 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:37.170028 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169609 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169612 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169614 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169617 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169619 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169621 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169624 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169627 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169629 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169632 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169634 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169636 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169638 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169641 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169643 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169647 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169650 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169652 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169655 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169657 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:37.170511 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169660 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169662 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169673 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169676 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169678 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169681 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169683 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169686 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169688 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169691 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169693 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169696 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169699 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169701 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169703 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169706 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169709 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169713 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169715 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:37.170988 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169720 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169723 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169727 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169729 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169732 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169735 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169737 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169741 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169744 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169748 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169751 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169753 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169756 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169759 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169761 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169764 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169766 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169769 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169772 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:37.171470 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.169774 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170449 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170456 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170458 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170461 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170464 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170467 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170470 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170473 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170475 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170478 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170480 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170483 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170487 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170490 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170492 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170495 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170497 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170500 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170502 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:37.171950 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170505 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170508 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170511 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170513 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170516 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170519 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170523 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170527 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170530 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170533 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170536 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170539 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170541 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170545 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170547 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170550 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170552 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170555 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170558 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:37.172496 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170562 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170564 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170567 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170571 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170574 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170578 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170581 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170583 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170586 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170590 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170592 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170595 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170597 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170600 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170602 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170605 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170608 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170610 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170613 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:37.172970 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170615 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170618 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170620 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170623 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170625 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170628 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170630 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170633 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170635 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170638 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170640 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170643 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170645 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170648 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170651 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170654 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170656 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170659 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170661 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170664 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:37.173465 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170666 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170669 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170671 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170674 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170676 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170680 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170682 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170685 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170689 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170692 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170695 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170698 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170700 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170703 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170705 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170707 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170710 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170713 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170715 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:37.173991 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170717 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170720 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.170722 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171070 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171076 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171080 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171083 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171086 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171089 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171091 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171094 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171097 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171100 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171102 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171105 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171107 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171110 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171114 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:37.174462 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171117 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171120 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171123 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171126 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171129 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171132 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171135 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171137 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171140 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171143 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171146 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171148 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171151 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171154 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171156 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171158 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171161 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171164 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171167 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171169 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:37.174912 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171172 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171174 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171177 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171179 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171182 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171184 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171187 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171190 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171192 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171195 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171197 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171200 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171202 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171205 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171207 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171209 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171212 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171214 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171217 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171219 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:37.175411 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171222 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171224 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171227 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171229 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171231 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171234 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171236 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171238 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171241 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171243 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171246 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171248 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171251 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171253 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171255 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171260 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171263 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171269 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171272 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171274 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:37.175904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171277 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171279 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171282 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171284 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171287 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171289 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171292 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171294 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171297 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171299 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171302 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171304 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171307 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171309 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171312 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171314 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171317 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171319 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171322 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171324 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:37.176404 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171326 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171329 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171332 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.171334 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171410 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171442 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171448 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171452 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171456 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171461 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171465 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171468 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171471 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171474 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171478 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171481 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171484 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171487 2580 flags.go:64] FLAG: --cgroup-root="" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171489 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171492 2580 flags.go:64] FLAG: --client-ca-file="" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171495 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171498 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171501 2580 flags.go:64] FLAG: --cluster-domain="" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171504 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 18:45:37.176883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171507 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171510 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171513 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171517 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171520 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171523 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171526 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171528 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171531 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171534 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171539 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171542 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171545 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171548 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171551 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171554 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171556 2580 flags.go:64] FLAG: --enable-server="true" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171563 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171568 2580 flags.go:64] FLAG: --event-burst="100" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171571 2580 flags.go:64] FLAG: --event-qps="50" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171574 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171577 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171580 2580 flags.go:64] FLAG: --eviction-hard="" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171583 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 18:45:37.177506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171586 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171589 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171592 2580 flags.go:64] FLAG: --eviction-soft="" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171594 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171597 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171600 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171603 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171605 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171608 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171611 2580 flags.go:64] FLAG: --feature-gates="" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171614 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171617 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171620 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171623 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171626 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171629 2580 flags.go:64] FLAG: --help="false" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171631 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171634 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171638 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171641 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171644 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171648 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171651 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 18:45:37.178084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171653 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171656 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171660 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171664 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171667 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171669 2580 flags.go:64] FLAG: --kube-reserved="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171672 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171675 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171678 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171680 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171683 2580 flags.go:64] FLAG: --lock-file="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171686 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171689 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171691 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171697 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171699 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171702 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171705 2580 flags.go:64] FLAG: --logging-format="text" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171708 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171711 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171714 2580 flags.go:64] FLAG: --manifest-url="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171716 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171720 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171723 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171727 2580 flags.go:64] FLAG: --max-pods="110" Sep 16 18:45:37.178735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171729 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171732 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171736 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171739 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171742 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171745 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171748 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171754 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171757 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171761 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171764 2580 flags.go:64] FLAG: --pod-cidr="" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171767 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171771 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171774 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171778 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171780 2580 flags.go:64] FLAG: --port="10250" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171783 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171786 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d17949e487671e12" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171789 2580 flags.go:64] FLAG: --qos-reserved="" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171792 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171795 2580 flags.go:64] FLAG: --register-node="true" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171797 2580 flags.go:64] FLAG: --register-with-taints="" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171801 2580 flags.go:64] FLAG: --registry-burst="10" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171803 2580 flags.go:64] FLAG: --registry-qps="5" Sep 16 18:45:37.179352 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171806 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171809 2580 flags.go:64] FLAG: --reserved-memory="" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171812 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171815 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171818 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171821 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171824 2580 flags.go:64] FLAG: --runonce="false" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171826 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171829 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171832 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171836 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171839 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171842 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171844 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171847 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171850 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171853 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171856 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171859 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171862 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171865 2580 flags.go:64] FLAG: --system-cgroups="" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171868 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171873 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171875 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171878 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 18:45:37.179963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171881 2580 flags.go:64] FLAG: --tls-min-version="" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171884 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171887 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171889 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171892 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171895 2580 flags.go:64] FLAG: --v="2" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171898 2580 flags.go:64] FLAG: --version="false" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171902 2580 flags.go:64] FLAG: --vmodule="" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171905 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.171908 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172028 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172032 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172035 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172040 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172043 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172047 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172049 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172053 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172056 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172059 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172062 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172064 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:37.180587 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172067 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172071 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172074 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172078 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172081 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172083 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172086 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172089 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172092 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172094 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172097 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172100 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172102 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172105 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172107 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172110 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172112 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172115 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172117 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172120 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:37.181138 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172122 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172125 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172127 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172129 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172132 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172135 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172138 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172141 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172144 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172147 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172150 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172152 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172155 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172158 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172160 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172163 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172165 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172168 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172170 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172173 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:37.181765 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172176 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172178 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172180 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172183 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172185 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172188 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172190 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172193 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172195 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172198 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172200 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172203 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172205 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172208 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172210 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172213 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172215 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172218 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172220 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172224 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:37.182271 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172226 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172228 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172231 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172233 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172236 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172238 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172241 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172243 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172246 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172248 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172251 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172253 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172256 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172258 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172261 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172264 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172268 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172270 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172273 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:37.182777 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172276 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172279 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172282 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172284 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172287 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172289 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172292 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.172294 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.173062 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.182801 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.182816 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.182833 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.182839 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182891 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:37.183241 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182896 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182900 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182904 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182908 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182911 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182914 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182917 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182920 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182923 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182925 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182928 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182931 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182933 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182936 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182938 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182941 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182944 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182946 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182949 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:37.183659 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182951 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182954 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182956 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182959 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182961 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182964 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182966 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182969 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182972 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182976 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182978 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182981 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182983 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182986 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182988 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182991 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182993 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182996 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.182998 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183001 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:37.184452 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183003 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183007 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183011 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183015 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183018 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183021 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183024 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183027 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183029 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183032 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183034 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183037 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183039 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183042 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183044 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183046 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183049 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183052 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183054 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:37.184986 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183057 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183060 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183063 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183066 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183069 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183071 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183074 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183076 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183079 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183081 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183084 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183086 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183089 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183091 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183094 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183096 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183099 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183101 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183104 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183106 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:37.185479 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183109 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183112 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183114 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183116 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183119 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183121 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183124 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183126 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183129 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183131 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183134 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183136 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183139 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183142 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183144 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183147 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183150 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183152 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183155 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:37.185953 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183157 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.183162 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183269 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183274 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183277 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183280 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183283 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183286 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183288 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183291 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183294 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183296 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183298 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183301 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183304 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:37.186429 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183308 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183310 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183313 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183315 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183318 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183320 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183322 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183325 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183327 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183332 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183335 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183337 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183341 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183344 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183347 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183350 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183352 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183355 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183357 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:37.186817 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183360 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183362 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183365 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183367 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183369 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183372 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183374 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183392 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183394 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183396 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183399 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183401 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183404 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183407 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183409 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183411 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183414 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183416 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183419 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:37.187277 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183421 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183424 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183426 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183429 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183431 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183434 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183436 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183439 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183441 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183444 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183447 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183449 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183452 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183454 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183457 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183459 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183462 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183464 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183467 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183469 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:37.187857 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183471 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183474 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183476 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183479 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183481 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183483 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183486 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183488 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183490 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183493 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183495 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183498 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183500 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183503 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183506 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183508 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183511 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183513 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183516 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183518 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:37.188363 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183521 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183523 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183527 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183531 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183534 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183537 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183539 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:37.183542 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.183546 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.184066 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.186569 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.187574 2580 server.go:1019] "Starting client certificate rotation" Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.187677 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:37.188864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.188493 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:37.191213 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.191200 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:37.192254 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.192238 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:37.208198 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.208183 2580 log.go:25] "Validated CRI v1 runtime API" Sep 16 18:45:37.211515 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.211502 2580 log.go:25] "Validated CRI v1 image API" Sep 16 18:45:37.212340 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.212322 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 18:45:37.216064 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.216048 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ccf492e1-aebc-4f9d-aa3e-c569be7f4943:/dev/nvme0n1p3 d46b385b-9e7b-459c-84d5-439dfe64b22f:/dev/nvme0n1p4] Sep 16 18:45:37.216128 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.216064 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 18:45:37.221520 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.221411 2580 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:37.220943081 +0000 UTC m=+0.343790577 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127060 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2162eea19ec51c7620551cd85fcf4e SystemUUID:ec2162ee-a19e-c51c-7620-551cd85fcf4e BootID:7b1411a3-393c-46c9-81e9-4bf70bd0bea3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:ab:0e:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:ab:0e:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:dd:7f:88:dc:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 18:45:37.221520 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.221512 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 18:45:37.221624 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.221575 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 18:45:37.221928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.221910 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:37.225294 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.225274 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 18:45:37.225445 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.225296 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 18:45:37.226188 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.226178 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 18:45:37.226219 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.226191 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 18:45:37.226219 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.226199 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:37.226219 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.226214 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:37.227427 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.227417 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 18:45:37.233233 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233221 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:37.233347 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233337 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 18:45:37.233390 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233365 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 16 18:45:37.233418 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233402 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 18:45:37.233496 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233486 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 18:45:37.233529 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233502 2580 kubelet.go:398] "Adding apiserver pod source" Sep 16 18:45:37.233557 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.233541 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 18:45:37.234103 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.234090 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:37.234165 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.234110 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:37.238870 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.238836 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 18:45:37.242122 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.242103 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 18:45:37.242221 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.242151 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 18:45:37.242221 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.242152 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 18:45:37.242221 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.242183 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 18:45:37.244062 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244048 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 18:45:37.244105 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244069 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 18:45:37.244105 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244076 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 18:45:37.244105 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244084 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 18:45:37.244105 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244096 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 18:45:37.244105 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244101 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 18:45:37.244240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244114 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 18:45:37.244240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244119 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 18:45:37.244240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244125 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 18:45:37.244240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244130 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 18:45:37.244240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244139 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 18:45:37.244240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.244144 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 18:45:37.245025 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.245015 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 18:45:37.245025 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.245026 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 18:45:37.247427 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.247410 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 18:45:37.249737 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.249722 2580 server.go:1268] "Started kubelet" Sep 16 18:45:37.249793 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.249768 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 18:45:37.249899 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.249859 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 18:45:37.249979 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.249920 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 18:45:37.250479 ip-10-0-128-157 systemd[1]: Started Kubernetes Kubelet. Sep 16 18:45:37.252173 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.252158 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 16 18:45:37.253743 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.253723 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 18:45:37.253810 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.253759 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 18:45:37.254158 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.254145 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9fvv6" Sep 16 18:45:37.259925 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.259910 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:37.260610 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.260596 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 18:45:37.260702 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.260684 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.261359 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261342 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 18:45:37.261359 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261360 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 18:45:37.261359 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261341 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 18:45:37.261581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261413 2580 factory.go:55] Registering systemd factory Sep 16 18:45:37.261581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261464 2580 factory.go:223] Registration of the systemd container factory successfully Sep 16 18:45:37.261581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261491 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 16 18:45:37.261581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261500 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 16 18:45:37.261850 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261690 2580 factory.go:153] Registering CRI-O factory Sep 16 18:45:37.261850 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261704 2580 factory.go:223] Registration of the crio container factory successfully Sep 16 18:45:37.261850 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261833 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 18:45:37.262138 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261854 2580 factory.go:103] Registering Raw factory Sep 16 18:45:37.262138 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.261890 2580 manager.go:1196] Started watching for new ooms in manager Sep 16 18:45:37.262449 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.262433 2580 manager.go:349] Starting recovery of all containers Sep 16 18:45:37.263677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.262436 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9fvv6" Sep 16 18:45:37.264231 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.264207 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 18:45:37.264787 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.264263 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 18:45:37.266131 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.264925 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-157.ec2.internal.18d5e1791d5cb6fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-157.ec2.internal,UID:ip-10-0-128-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-157.ec2.internal,},FirstTimestamp:2026-09-16 18:45:37.24954803 +0000 UTC m=+0.372395527,LastTimestamp:2026-09-16 18:45:37.24954803 +0000 UTC m=+0.372395527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-157.ec2.internal,}" Sep 16 18:45:37.273654 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.273631 2580 manager.go:354] Recovery completed Sep 16 18:45:37.278281 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.278263 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.282955 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.282942 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.283022 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.282970 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.283022 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.282983 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.283493 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.283475 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 18:45:37.283493 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.283492 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 18:45:37.283602 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.283491 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.283602 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.283509 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:37.284222 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.284205 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.284284 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.284236 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.284284 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.284246 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.284942 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.284880 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-157.ec2.internal.18d5e1791f5a7a37 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-157.ec2.internal,UID:ip-10-0-128-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-157.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-157.ec2.internal,},FirstTimestamp:2026-09-16 18:45:37.282955831 +0000 UTC m=+0.405803328,LastTimestamp:2026-09-16 18:45:37.282955831 +0000 UTC m=+0.405803328,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-157.ec2.internal,}" Sep 16 18:45:37.285548 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.285533 2580 policy_none.go:49] "None policy: Start" Sep 16 18:45:37.285548 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.285550 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 18:45:37.285635 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.285560 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 18:45:37.287533 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.287518 2580 policy_none.go:47] "Start" Sep 16 18:45:37.326931 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.326918 2580 manager.go:337] "Starting Device Plugin manager" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.326952 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.326961 2580 server.go:87] "Starting device plugin registration server" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.327151 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.327161 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.327269 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.327342 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.327351 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.327758 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 18:45:37.341768 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.327797 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.394084 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.394057 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 18:45:37.395285 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.395268 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 18:45:37.395360 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.395318 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 18:45:37.395360 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.395337 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 18:45:37.395475 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.395368 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 18:45:37.398404 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.398368 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:37.427755 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.427719 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.428532 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.428514 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.428626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.428538 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.428626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.428548 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.428626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.428565 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.437980 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.437966 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.438041 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.437986 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-157.ec2.internal\": node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.451616 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.451599 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.495644 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.495622 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal"] Sep 16 18:45:37.495723 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.495702 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.496409 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.496395 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.496471 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.496424 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.496471 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.496434 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.498717 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.498705 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.498830 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.498817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.498872 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.498844 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.499332 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.499304 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.499332 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.499326 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.499332 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.499335 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.499489 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.499420 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.499489 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.499442 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.499489 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.499451 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.501470 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.501456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.501515 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.501481 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:37.502134 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.502121 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:37.502205 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.502143 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:37.502205 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.502152 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:37.524880 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.524862 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-157.ec2.internal\" not found" node="ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.529393 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.529368 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-157.ec2.internal\" not found" node="ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.551801 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.551781 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.563869 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.563846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4df45dbfe5c84788f79ebe461f5c39c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal\" (UID: \"d4df45dbfe5c84788f79ebe461f5c39c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.563948 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.563878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4df45dbfe5c84788f79ebe461f5c39c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal\" (UID: \"d4df45dbfe5c84788f79ebe461f5c39c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.563948 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.563902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b301d645c57b142bcde84cc94cf8ddc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-157.ec2.internal\" (UID: \"b301d645c57b142bcde84cc94cf8ddc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.652562 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.652426 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.664183 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.664162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b301d645c57b142bcde84cc94cf8ddc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-157.ec2.internal\" (UID: \"b301d645c57b142bcde84cc94cf8ddc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.664240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.664188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4df45dbfe5c84788f79ebe461f5c39c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal\" (UID: \"d4df45dbfe5c84788f79ebe461f5c39c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.664240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.664203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4df45dbfe5c84788f79ebe461f5c39c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal\" (UID: \"d4df45dbfe5c84788f79ebe461f5c39c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.664336 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.664237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4df45dbfe5c84788f79ebe461f5c39c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal\" (UID: \"d4df45dbfe5c84788f79ebe461f5c39c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.664336 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.664244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b301d645c57b142bcde84cc94cf8ddc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-157.ec2.internal\" (UID: \"b301d645c57b142bcde84cc94cf8ddc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.664336 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.664278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4df45dbfe5c84788f79ebe461f5c39c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal\" (UID: \"d4df45dbfe5c84788f79ebe461f5c39c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.753544 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.753498 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.826911 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.826890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.831607 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:37.831560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" Sep 16 18:45:37.854062 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.854033 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:37.954594 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:37.954568 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.055057 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.055031 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.155587 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.155564 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.187030 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.187007 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 18:45:38.187404 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.187143 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:45:38.256146 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.256122 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.273115 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.273089 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:37 +0000 UTC" deadline="2028-04-29 08:58:13.458147985 +0000 UTC" Sep 16 18:45:38.273115 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.273110 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14174h12m35.185040362s" Sep 16 18:45:38.356448 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.356397 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.443740 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:38.443714 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4df45dbfe5c84788f79ebe461f5c39c.slice/crio-16b6a6802bf95dff362381b0212b07c14667132ad69a75ab53122929026c02d9 WatchSource:0}: Error finding container 16b6a6802bf95dff362381b0212b07c14667132ad69a75ab53122929026c02d9: Status 404 returned error can't find the container with id 16b6a6802bf95dff362381b0212b07c14667132ad69a75ab53122929026c02d9 Sep 16 18:45:38.446988 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.446969 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:45:38.456965 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.456949 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.512323 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:38.512305 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb301d645c57b142bcde84cc94cf8ddc5.slice/crio-bed30cea8e10f098cfc7ff9d0e97823561e56e265cf4d2956c79b9cd2ab217d7 WatchSource:0}: Error finding container bed30cea8e10f098cfc7ff9d0e97823561e56e265cf4d2956c79b9cd2ab217d7: Status 404 returned error can't find the container with id bed30cea8e10f098cfc7ff9d0e97823561e56e265cf4d2956c79b9cd2ab217d7 Sep 16 18:45:38.557637 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.557614 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.641404 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.641346 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:38.658283 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:38.658263 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-157.ec2.internal\" not found" Sep 16 18:45:38.663012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.662998 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:38.741806 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.741787 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:38.761704 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.761685 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" Sep 16 18:45:38.773054 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.773038 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:38.773918 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.773906 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" Sep 16 18:45:38.780017 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:38.780004 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:39.127964 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.127938 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:39.234864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.234842 2580 apiserver.go:52] "Watching apiserver" Sep 16 18:45:39.239757 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.239735 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 18:45:39.240859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.240836 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jxqsh","kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal","openshift-dns/node-resolver-vshfx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal","openshift-multus/multus-additional-cni-plugins-pg56t","openshift-multus/network-metrics-daemon-8tzg8","openshift-network-diagnostics/network-check-target-t6x4g","openshift-network-operator/iptables-alerter-rlpsk","kube-system/konnectivity-agent-sgdnp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7","openshift-cluster-node-tuning-operator/tuned-p5lx8","openshift-image-registry/node-ca-dw5h8","openshift-multus/multus-gk2dp","openshift-ovn-kubernetes/ovnkube-node-gz585"] Sep 16 18:45:39.243075 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.243051 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.245881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.245107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.245881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.245291 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 18:45:39.245881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.245606 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.245881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.245671 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.245881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.245806 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dv4x7\"" Sep 16 18:45:39.246794 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.246778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.247065 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.247047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.247128 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.247110 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 18:45:39.247306 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.247291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.247429 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.247415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h7jwr\"" Sep 16 18:45:39.247468 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.247429 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 18:45:39.248989 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.248967 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.249174 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.249160 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6fx9s\"" Sep 16 18:45:39.249402 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.249388 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.249650 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.249634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.251303 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251276 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251584 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251600 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251673 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g6m2q\"" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251584 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.251960 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 18:45:39.252121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.252038 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 18:45:39.253341 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.253293 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 18:45:39.253452 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.253388 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vjwhx\"" Sep 16 18:45:39.253507 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.253481 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 18:45:39.254219 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.254203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.254290 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.254272 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:39.256950 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.256934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.258822 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.258798 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 18:45:39.259012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.258994 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zhd94\"" Sep 16 18:45:39.259097 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.259070 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 18:45:39.259887 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.259875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:39.267282 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.265911 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:39.267282 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.266003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.267282 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.266008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.267282 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.266643 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:39.267282 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.267064 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:39.268093 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.268075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.270513 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.269870 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.270513 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.269900 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.270513 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.270113 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 18:45:39.270513 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.270117 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mdgjd\"" Sep 16 18:45:39.270513 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.270401 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-98lm8\"" Sep 16 18:45:39.270749 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.270640 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.270954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.270939 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.272976 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.272952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.274627 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-node-log\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.274720 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-cni-netd\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.274720 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9922bad3-6630-46c7-8413-78a787ea7000-agent-certs\") pod \"konnectivity-agent-sgdnp\" (UID: \"9922bad3-6630-46c7-8413-78a787ea7000\") " pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.274720 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274618 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 18:45:39.274860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fc4\" (UniqueName: \"kubernetes.io/projected/e50d547c-882d-4029-96ab-6d960ba6b3c1-kube-api-access-f2fc4\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.274860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85ed07c9-cec1-4a9f-92af-f3045a969eb0-tmp\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.274860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-cni-multus\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.274860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-cni-bin\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275032 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274857 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hssbl\"" Sep 16 18:45:39.275032 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274866 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 18:45:39.275032 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfsg\" (UniqueName: \"kubernetes.io/projected/85ed07c9-cec1-4a9f-92af-f3045a969eb0-kube-api-access-bnfsg\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275032 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adf41b5e-ddca-4667-903e-1c56061d678f-cni-binary-copy\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275032 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.274988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/adf41b5e-ddca-4667-903e-1c56061d678f-multus-daemon-config\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275032 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-device-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50d547c-882d-4029-96ab-6d960ba6b3c1-hosts-file\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275036 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-socket-dir-parent\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-k8s-cni-cncf-io\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-multus-certs\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfz6\" (UniqueName: \"kubernetes.io/projected/593cdcf1-d960-4800-a58f-46e20476cb32-kube-api-access-8nfz6\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysconfig\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-run\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-var-lib-kubelet\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-host\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppxq\" (UniqueName: \"kubernetes.io/projected/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-kube-api-access-zppxq\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-etc-kubernetes\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-ovn\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-lib-modules\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-cni-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-log-socket\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-sys-fs\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-system-cni-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-kubelet\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-slash\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovn-node-metrics-cert\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysctl-conf\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.275736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-sys\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-netns\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-systemd\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovnkube-config\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovnkube-script-lib\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-systemd\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-cni-bin\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-kubelet\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-hostroot\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.275986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-systemd-units\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-iptables-alerter-script\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-dbus\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.276484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2jk\" (UniqueName: \"kubernetes.io/projected/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-kube-api-access-2m2jk\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tvf\" (UniqueName: \"kubernetes.io/projected/adf41b5e-ddca-4667-903e-1c56061d678f-kube-api-access-x6tvf\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-var-lib-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-kubernetes\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysctl-d\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-tuned\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-system-cni-dir\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-env-overrides\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfm4\" (UniqueName: \"kubernetes.io/projected/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-kube-api-access-5nfm4\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9922bad3-6630-46c7-8413-78a787ea7000-konnectivity-ca\") pod \"konnectivity-agent-sgdnp\" (UID: \"9922bad3-6630-46c7-8413-78a787ea7000\") " pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-host-slash\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50d547c-882d-4029-96ab-6d960ba6b3c1-tmp-dir\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cnibin\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-modprobe-d\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.277193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-cnibin\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-conf-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-run-netns\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-kubelet-config\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-socket-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-etc-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkcl\" (UniqueName: \"kubernetes.io/projected/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-kube-api-access-5fkcl\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-registration-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.277938 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.276930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-etc-selinux\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.281291 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.281160 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:39.299391 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.299354 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x9wwt" Sep 16 18:45:39.307955 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.307933 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x9wwt" Sep 16 18:45:39.362984 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.362963 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 18:45:39.377443 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-sys\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.377552 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-netns\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.377552 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-systemd\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.377552 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-netns\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.377679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-sys\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.377679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovnkube-config\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.377679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovnkube-script-lib\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.377813 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.377813 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d752bc5-c177-4081-8396-b249b8eacd3f-host\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.377813 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-systemd\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.377813 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.377813 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-systemd\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-systemd\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.377839 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-cni-bin\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.377907 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.87788846 +0000 UTC m=+3.000735960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-kubelet\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-cni-bin\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-hostroot\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.377993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-hostroot\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.378048 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-systemd-units\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-systemd-units\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-iptables-alerter-script\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-kubelet\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovnkube-config\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-dbus\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-dbus\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2jk\" (UniqueName: \"kubernetes.io/projected/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-kube-api-access-2m2jk\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.378474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tvf\" (UniqueName: \"kubernetes.io/projected/adf41b5e-ddca-4667-903e-1c56061d678f-kube-api-access-x6tvf\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-var-lib-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-kubernetes\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysctl-d\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-tuned\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-system-cni-dir\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-iptables-alerter-script\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-env-overrides\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfm4\" (UniqueName: \"kubernetes.io/projected/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-kube-api-access-5nfm4\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysctl-d\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.378968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9922bad3-6630-46c7-8413-78a787ea7000-konnectivity-ca\") pod \"konnectivity-agent-sgdnp\" (UID: \"9922bad3-6630-46c7-8413-78a787ea7000\") " pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-host-slash\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.379041 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50d547c-882d-4029-96ab-6d960ba6b3c1-tmp-dir\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cnibin\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-modprobe-d\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379092 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-cnibin\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-conf-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.379145 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-run-netns\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.379195 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.879179738 +0000 UTC m=+3.002027234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-kubelet-config\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-socket-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-system-cni-dir\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-etc-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-etc-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkcl\" (UniqueName: \"kubernetes.io/projected/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-kube-api-access-5fkcl\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.379677 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379419 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-kubelet-config\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-registration-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-etc-selinux\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-var-lib-openvswitch\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d752bc5-c177-4081-8396-b249b8eacd3f-serviceca\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-node-log\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9922bad3-6630-46c7-8413-78a787ea7000-konnectivity-ca\") pod \"konnectivity-agent-sgdnp\" (UID: \"9922bad3-6630-46c7-8413-78a787ea7000\") " pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-kubernetes\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-cni-netd\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovnkube-script-lib\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-socket-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9922bad3-6630-46c7-8413-78a787ea7000-agent-certs\") pod \"konnectivity-agent-sgdnp\" (UID: \"9922bad3-6630-46c7-8413-78a787ea7000\") " pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-host-slash\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fc4\" (UniqueName: \"kubernetes.io/projected/e50d547c-882d-4029-96ab-6d960ba6b3c1-kube-api-access-f2fc4\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85ed07c9-cec1-4a9f-92af-f3045a969eb0-tmp\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-modprobe-d\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.380482 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-conf-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-cni-multus\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cnibin\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-var-lib-cni-multus\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-cni-bin\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-node-log\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-cni-netd\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-cni-bin\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-cnibin\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-registration-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.379988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50d547c-882d-4029-96ab-6d960ba6b3c1-tmp-dir\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-run-netns\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfsg\" (UniqueName: \"kubernetes.io/projected/85ed07c9-cec1-4a9f-92af-f3045a969eb0-kube-api-access-bnfsg\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-etc-selinux\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adf41b5e-ddca-4667-903e-1c56061d678f-cni-binary-copy\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/adf41b5e-ddca-4667-903e-1c56061d678f-multus-daemon-config\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-device-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.381192 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50d547c-882d-4029-96ab-6d960ba6b3c1-hosts-file\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-socket-dir-parent\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-k8s-cni-cncf-io\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-multus-certs\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfz6\" (UniqueName: \"kubernetes.io/projected/593cdcf1-d960-4800-a58f-46e20476cb32-kube-api-access-8nfz6\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysconfig\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-run\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-var-lib-kubelet\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-host\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggdq\" (UniqueName: \"kubernetes.io/projected/3d752bc5-c177-4081-8396-b249b8eacd3f-kube-api-access-fggdq\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppxq\" (UniqueName: \"kubernetes.io/projected/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-kube-api-access-zppxq\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-etc-kubernetes\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-ovn\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-env-overrides\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adf41b5e-ddca-4667-903e-1c56061d678f-cni-binary-copy\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-lib-modules\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-cni-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-log-socket\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-socket-dir-parent\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-sys-fs\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-k8s-cni-cncf-io\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-run\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-var-lib-kubelet\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysconfig\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-host-run-multus-certs\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-system-cni-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-run-ovn\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-kubelet\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-device-dir\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-multus-cni-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.382928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50d547c-882d-4029-96ab-6d960ba6b3c1-hosts-file\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.380995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/adf41b5e-ddca-4667-903e-1c56061d678f-multus-daemon-config\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-lib-modules\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-host\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-slash\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-log-socket\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-etc-kubernetes\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovn-node-metrics-cert\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-kubelet\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysctl-conf\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adf41b5e-ddca-4667-903e-1c56061d678f-system-cni-dir\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-host-slash\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-sys-fs\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.383626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.381442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-sysctl-conf\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.384416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.384235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85ed07c9-cec1-4a9f-92af-f3045a969eb0-tmp\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.384416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.384262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85ed07c9-cec1-4a9f-92af-f3045a969eb0-etc-tuned\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.384416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.384339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-ovn-node-metrics-cert\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.385914 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.384830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9922bad3-6630-46c7-8413-78a787ea7000-agent-certs\") pod \"konnectivity-agent-sgdnp\" (UID: \"9922bad3-6630-46c7-8413-78a787ea7000\") " pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.386164 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.385962 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:39.386164 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.385982 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:39.386164 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.385996 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:39.386164 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.386056 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.88603593 +0000 UTC m=+3.008883428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:39.391116 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.391022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfz6\" (UniqueName: \"kubernetes.io/projected/593cdcf1-d960-4800-a58f-46e20476cb32-kube-api-access-8nfz6\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.391596 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.391577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkcl\" (UniqueName: \"kubernetes.io/projected/6c6655a8-ef36-4cd3-a422-1e628e7d4aa4-kube-api-access-5fkcl\") pod \"iptables-alerter-rlpsk\" (UID: \"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4\") " pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.391880 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.391859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tvf\" (UniqueName: \"kubernetes.io/projected/adf41b5e-ddca-4667-903e-1c56061d678f-kube-api-access-x6tvf\") pod \"multus-gk2dp\" (UID: \"adf41b5e-ddca-4667-903e-1c56061d678f\") " pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.391934 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.391899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfm4\" (UniqueName: \"kubernetes.io/projected/6f03d949-aa22-4e8b-88d8-db70fd67f5b4-kube-api-access-5nfm4\") pod \"ovnkube-node-gz585\" (UID: \"6f03d949-aa22-4e8b-88d8-db70fd67f5b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.391964 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.391920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2jk\" (UniqueName: \"kubernetes.io/projected/6c9ee36a-4dc3-4b11-a07a-1a1f18480286-kube-api-access-2m2jk\") pod \"aws-ebs-csi-driver-node-5r2t7\" (UID: \"6c9ee36a-4dc3-4b11-a07a-1a1f18480286\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.392818 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.392714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppxq\" (UniqueName: \"kubernetes.io/projected/4cd8ecf3-a922-4d6e-a312-5a775bb163c0-kube-api-access-zppxq\") pod \"multus-additional-cni-plugins-pg56t\" (UID: \"4cd8ecf3-a922-4d6e-a312-5a775bb163c0\") " pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.393073 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.393059 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfsg\" (UniqueName: \"kubernetes.io/projected/85ed07c9-cec1-4a9f-92af-f3045a969eb0-kube-api-access-bnfsg\") pod \"tuned-p5lx8\" (UID: \"85ed07c9-cec1-4a9f-92af-f3045a969eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.393298 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.393276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fc4\" (UniqueName: \"kubernetes.io/projected/e50d547c-882d-4029-96ab-6d960ba6b3c1-kube-api-access-f2fc4\") pod \"node-resolver-vshfx\" (UID: \"e50d547c-882d-4029-96ab-6d960ba6b3c1\") " pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.400064 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.400024 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" event={"ID":"d4df45dbfe5c84788f79ebe461f5c39c","Type":"ContainerStarted","Data":"16b6a6802bf95dff362381b0212b07c14667132ad69a75ab53122929026c02d9"} Sep 16 18:45:39.401678 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.401658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" event={"ID":"b301d645c57b142bcde84cc94cf8ddc5","Type":"ContainerStarted","Data":"bed30cea8e10f098cfc7ff9d0e97823561e56e265cf4d2956c79b9cd2ab217d7"} Sep 16 18:45:39.481605 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.481580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d752bc5-c177-4081-8396-b249b8eacd3f-serviceca\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.481716 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.481613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggdq\" (UniqueName: \"kubernetes.io/projected/3d752bc5-c177-4081-8396-b249b8eacd3f-kube-api-access-fggdq\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.481716 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.481644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d752bc5-c177-4081-8396-b249b8eacd3f-host\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.481716 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.481702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d752bc5-c177-4081-8396-b249b8eacd3f-host\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.482849 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.482825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d752bc5-c177-4081-8396-b249b8eacd3f-serviceca\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.489732 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.489712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggdq\" (UniqueName: \"kubernetes.io/projected/3d752bc5-c177-4081-8396-b249b8eacd3f-kube-api-access-fggdq\") pod \"node-ca-dw5h8\" (UID: \"3d752bc5-c177-4081-8396-b249b8eacd3f\") " pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.555646 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.555609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" Sep 16 18:45:39.572179 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.572152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk2dp" Sep 16 18:45:39.586783 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.586760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vshfx" Sep 16 18:45:39.592282 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.592267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:45:39.598458 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.598439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pg56t" Sep 16 18:45:39.605867 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.605849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:45:39.613173 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.613156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rlpsk" Sep 16 18:45:39.619363 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.619339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" Sep 16 18:45:39.624714 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.624695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw5h8" Sep 16 18:45:39.885836 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.885800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:39.885996 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.885850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:39.885996 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.885949 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:39.885996 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.885985 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:39.886145 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.886011 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.885992854 +0000 UTC m=+4.008840349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:39.886145 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.886031 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.886017614 +0000 UTC m=+4.008865114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:39.986361 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:39.986331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:39.986523 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.986492 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:39.986523 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.986509 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:39.986523 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.986519 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:39.986677 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:39.986569 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.986554682 +0000 UTC m=+4.109402164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:40.271481 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:40.271453 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d752bc5_c177_4081_8396_b249b8eacd3f.slice/crio-db1760b4bcdf6cce0550f2d6284f4209035c639780d1e7f9960599f89a9893df WatchSource:0}: Error finding container db1760b4bcdf6cce0550f2d6284f4209035c639780d1e7f9960599f89a9893df: Status 404 returned error can't find the container with id db1760b4bcdf6cce0550f2d6284f4209035c639780d1e7f9960599f89a9893df Sep 16 18:45:40.273599 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:45:40.273549 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd8ecf3_a922_4d6e_a312_5a775bb163c0.slice/crio-910c8eb086c4d042339fe8be7408e9f1386ae943b74f3140e974df5956615a8a WatchSource:0}: Error finding container 910c8eb086c4d042339fe8be7408e9f1386ae943b74f3140e974df5956615a8a: Status 404 returned error can't find the container with id 910c8eb086c4d042339fe8be7408e9f1386ae943b74f3140e974df5956615a8a Sep 16 18:45:40.312015 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.311989 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:39 +0000 UTC" deadline="2028-06-17 09:25:39.500108774 +0000 UTC" Sep 16 18:45:40.312015 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.312014 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15350h39m59.188097924s" Sep 16 18:45:40.396517 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.396497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:40.396641 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.396617 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:40.404545 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.404520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" event={"ID":"6c9ee36a-4dc3-4b11-a07a-1a1f18480286","Type":"ContainerStarted","Data":"3d92c3db477bb10ad4bdf10c3cd2b46ec3baac39f61fee996a5fad15b20acc0f"} Sep 16 18:45:40.405563 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.405544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw5h8" event={"ID":"3d752bc5-c177-4081-8396-b249b8eacd3f","Type":"ContainerStarted","Data":"db1760b4bcdf6cce0550f2d6284f4209035c639780d1e7f9960599f89a9893df"} Sep 16 18:45:40.412353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.412332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"49bccaec9056830724f1260ca2070024207c67ce0c77507363e1bf1625790ea4"} Sep 16 18:45:40.414234 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.414196 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rlpsk" event={"ID":"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4","Type":"ContainerStarted","Data":"a7521e2286e77320ce3fd17f2577dc14ec9fbd4607308808cb93375a27f657a8"} Sep 16 18:45:40.415254 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.415232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vshfx" event={"ID":"e50d547c-882d-4029-96ab-6d960ba6b3c1","Type":"ContainerStarted","Data":"cc46335643f8f85e4a9933e6aff7785217aabcc2410b0679c9f8df10b83e2c6c"} Sep 16 18:45:40.416940 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.416919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk2dp" event={"ID":"adf41b5e-ddca-4667-903e-1c56061d678f","Type":"ContainerStarted","Data":"991cf4815d8b88364c3625b49c38b8169e516d3d72c1eced2c28cfe513d9dbd7"} Sep 16 18:45:40.419308 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.419289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgdnp" event={"ID":"9922bad3-6630-46c7-8413-78a787ea7000","Type":"ContainerStarted","Data":"e03b1726682beecfa6c7dbe2740dac9c14b650b8316d15f8dcaf559962615e53"} Sep 16 18:45:40.420372 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.420352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerStarted","Data":"910c8eb086c4d042339fe8be7408e9f1386ae943b74f3140e974df5956615a8a"} Sep 16 18:45:40.423600 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.423580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" event={"ID":"b301d645c57b142bcde84cc94cf8ddc5","Type":"ContainerStarted","Data":"067ae32c65990e41c25ad602bf03f2b2777bde7378b491f56575805ce59c0979"} Sep 16 18:45:40.425195 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.425172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" event={"ID":"85ed07c9-cec1-4a9f-92af-f3045a969eb0","Type":"ContainerStarted","Data":"f04c3ee59a0f4ee498cf01a1aa210e7ebc7cf2aef627e66cbd5dc0a452456ec4"} Sep 16 18:45:40.450743 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.450595 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-157.ec2.internal" podStartSLOduration=2.450582054 podStartE2EDuration="2.450582054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:40.448455373 +0000 UTC m=+3.571302876" watchObservedRunningTime="2026-09-16 18:45:40.450582054 +0000 UTC m=+3.573429559" Sep 16 18:45:40.894583 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.894538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:40.894759 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.894609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:40.894759 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.894738 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:40.894888 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.894815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.894792819 +0000 UTC m=+6.017640318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:40.895226 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.895206 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:40.895299 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.895268 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.895251555 +0000 UTC m=+6.018099039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:40.995233 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:40.995192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:40.995476 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.995458 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:40.995544 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.995484 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:40.995544 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.995496 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:40.995658 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:40.995554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.995536246 +0000 UTC m=+6.118383737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:41.312699 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:41.312625 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:39 +0000 UTC" deadline="2028-05-22 18:13:24.826631054 +0000 UTC" Sep 16 18:45:41.312699 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:41.312689 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14735h27m43.513946407s" Sep 16 18:45:41.397532 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:41.397501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:41.397725 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:41.397639 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:41.397725 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:41.397680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:41.397830 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:41.397785 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:41.430820 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:41.430787 2580 generic.go:356] "Generic (PLEG): container finished" podID="d4df45dbfe5c84788f79ebe461f5c39c" containerID="adfd65cee473118fc1a873ccd6ca45e643212cdea8b09c3e9cc506ca7f625575" exitCode=0 Sep 16 18:45:41.431406 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:41.431365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" event={"ID":"d4df45dbfe5c84788f79ebe461f5c39c","Type":"ContainerDied","Data":"adfd65cee473118fc1a873ccd6ca45e643212cdea8b09c3e9cc506ca7f625575"} Sep 16 18:45:42.396406 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:42.396336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:42.396841 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:42.396485 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:42.451287 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:42.450584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" event={"ID":"d4df45dbfe5c84788f79ebe461f5c39c","Type":"ContainerStarted","Data":"8485b33f3463e765066420e3d44e391cbce1cefbc051b4753dc921d57b5a486c"} Sep 16 18:45:42.912337 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:42.912297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:42.912539 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:42.912358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:42.912539 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:42.912523 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:42.912643 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:42.912587 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.912568139 +0000 UTC m=+10.035415625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:42.913000 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:42.912977 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:42.913108 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:42.913035 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.913019374 +0000 UTC m=+10.035866862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:43.013452 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:43.013362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:43.013639 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:43.013621 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:43.013700 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:43.013647 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:43.013700 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:43.013662 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:43.013826 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:43.013811 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.013716696 +0000 UTC m=+10.136564184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:43.396251 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:43.396224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:43.396439 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:43.396361 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:43.396860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:43.396840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:43.396979 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:43.396949 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:44.396362 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:44.396327 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:44.396571 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:44.396502 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:45.397182 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:45.396649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:45.397182 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:45.396648 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:45.397182 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:45.396804 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:45.397182 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:45.396849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:46.395980 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:46.395943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:46.396185 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:46.396095 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:46.949834 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:46.949798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:46.950283 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:46.949856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:46.950283 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:46.949995 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:46.950283 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:46.950055 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:54.950034767 +0000 UTC m=+18.072882254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:46.950477 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:46.950458 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:46.950527 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:46.950494 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:54.95048378 +0000 UTC m=+18.073331277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:47.050690 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:47.050655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:47.051327 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:47.050900 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:47.051327 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:47.050956 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:47.051327 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:47.050971 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:47.051327 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:47.051048 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:55.05102896 +0000 UTC m=+18.173876446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:47.397544 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:47.397479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:47.398461 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:47.398424 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:47.398797 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:47.398768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:47.398908 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:47.398868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:48.396042 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:48.396003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:48.396664 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:48.396138 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:49.399211 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:49.399177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:49.399686 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:49.399177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:49.399686 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:49.399316 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:49.399686 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:49.399403 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:50.395923 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:50.395889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:50.396102 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:50.395992 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:51.396185 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:51.396156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:51.396671 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:51.396280 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:51.396671 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:51.396325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:51.396671 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:51.396413 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:52.395736 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:52.395709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:52.395936 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:52.395892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:53.396594 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:53.396563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:53.397058 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:53.396705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:53.397058 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:53.396761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:53.397058 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:53.396872 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:54.396168 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:54.396138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:54.396430 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:54.396251 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:55.012099 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:55.012067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:55.012579 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:55.012109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:55.012579 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.012237 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:55.012579 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.012287 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:55.012579 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.012302 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:11.012283581 +0000 UTC m=+34.135131079 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:55.012579 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.012321 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:11.012310401 +0000 UTC m=+34.135157885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:55.113242 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:55.113208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:55.113443 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.113341 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:55.113443 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.113359 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:55.113443 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.113370 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:55.113586 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.113447 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:11.113428279 +0000 UTC m=+34.236275767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:55.396133 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:55.395691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:55.396133 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.395816 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:55.396133 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:55.395882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:55.396133 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:55.395972 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:56.396289 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:56.396258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:56.396736 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:56.396368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:57.396234 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:57.396202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:57.396410 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:57.396304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:57.396410 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:57.396361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:57.396789 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:57.396475 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:58.395784 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:58.395763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:45:58.395872 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:58.395848 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:45:59.395903 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.395590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:45:59.396777 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.395686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:45:59.396777 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:59.395989 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:45:59.396777 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:45:59.396060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:45:59.501212 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.501171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" event={"ID":"6c9ee36a-4dc3-4b11-a07a-1a1f18480286","Type":"ContainerStarted","Data":"e0ff2489cdd10d97b1f2e829225d0ae728f661ec4cf3963ff8676307e928ac6a"} Sep 16 18:45:59.502859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.502833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw5h8" event={"ID":"3d752bc5-c177-4081-8396-b249b8eacd3f","Type":"ContainerStarted","Data":"3794baabd1a9b258f5fced994ce77eab321c14191929ace44647b2d62fb54181"} Sep 16 18:45:59.506531 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.506498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"9756192010da2ae4405b84b97f9a275b37d9ad50957a719abfb255da0929c8f2"} Sep 16 18:45:59.506645 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.506561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"4a9d8e4ec4668f5cd9704d91e1af034aca636cbcf09bbb02c49163ce9cb821f9"} Sep 16 18:45:59.506645 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.506595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"2483fc4951236932a4ea5a1fcb2e625ea09b078a2f61e5dee36b86810dce8d69"} Sep 16 18:45:59.506645 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.506640 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"a3126e5a17ad8329b14b379ba2ba9f7fbb9a8305f309d7d9c7ea8e791f97f343"} Sep 16 18:45:59.506797 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.506685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"8570162e23f6d613adb615c9d67e43607048df1485d6a517e4286509e0f5d942"} Sep 16 18:45:59.506797 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.506731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"507e54afdad71ccec71199decfedaeb099b424b27aa43313f8ee64224b194d10"} Sep 16 18:45:59.508077 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.508051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vshfx" event={"ID":"e50d547c-882d-4029-96ab-6d960ba6b3c1","Type":"ContainerStarted","Data":"46383a25e9ccbca06dec0db1261dfeacb0513f97e0e1b551456dd678166e8402"} Sep 16 18:45:59.509793 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.509768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk2dp" event={"ID":"adf41b5e-ddca-4667-903e-1c56061d678f","Type":"ContainerStarted","Data":"bd2a481ca2b0e18c5e7c2478e881047ba36bbc84cb1b0d9fb2853abd59bce460"} Sep 16 18:45:59.511321 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.511296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgdnp" event={"ID":"9922bad3-6630-46c7-8413-78a787ea7000","Type":"ContainerStarted","Data":"c6426875220ac9ddbbfc04c0a55ea505174bf891490c5c8a62a6a4cd215a98e7"} Sep 16 18:45:59.513866 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.513268 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cd8ecf3-a922-4d6e-a312-5a775bb163c0" containerID="fe0b34979fd9dca192cfa0840fa52990c43484e3bda87db4dfc0d63a9c76e5a8" exitCode=0 Sep 16 18:45:59.513866 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.513338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerDied","Data":"fe0b34979fd9dca192cfa0840fa52990c43484e3bda87db4dfc0d63a9c76e5a8"} Sep 16 18:45:59.516913 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.516892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" event={"ID":"85ed07c9-cec1-4a9f-92af-f3045a969eb0","Type":"ContainerStarted","Data":"5d96a904ad9b32fdc7b928be8c13494c44b636f0e115e65f98b14c71a7c6d173"} Sep 16 18:45:59.528339 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.528292 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-157.ec2.internal" podStartSLOduration=21.528251386 podStartE2EDuration="21.528251386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.472112737 +0000 UTC m=+5.594960249" watchObservedRunningTime="2026-09-16 18:45:59.528251386 +0000 UTC m=+22.651098893" Sep 16 18:45:59.557495 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.557456 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dw5h8" podStartSLOduration=4.376257388 podStartE2EDuration="22.557439005s" totalImagesPullingTime="18.181181617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:59.528658028 +0000 UTC m=+22.651505537" watchObservedRunningTime="2026-09-16 18:45:59.557439005 +0000 UTC m=+22.680286511" Sep 16 18:45:59.581256 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.581198 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p5lx8" podStartSLOduration=4.393800885 podStartE2EDuration="22.581186068s" totalImagesPullingTime="18.187385183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:59.558059465 +0000 UTC m=+22.680906971" watchObservedRunningTime="2026-09-16 18:45:59.581186068 +0000 UTC m=+22.704033573" Sep 16 18:45:59.581346 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.581327 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gk2dp" podStartSLOduration=4.355059254 podStartE2EDuration="22.581320851s" totalImagesPullingTime="18.226261597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:59.581078132 +0000 UTC m=+22.703925637" watchObservedRunningTime="2026-09-16 18:45:59.581320851 +0000 UTC m=+22.704168357" Sep 16 18:45:59.630622 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.630592 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vshfx" podStartSLOduration=4.457055266 podStartE2EDuration="22.63058205s" totalImagesPullingTime="18.173526784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:59.603487785 +0000 UTC m=+22.726335293" watchObservedRunningTime="2026-09-16 18:45:59.63058205 +0000 UTC m=+22.753429556" Sep 16 18:45:59.675911 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.675871 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sgdnp" podStartSLOduration=4.762292023 podStartE2EDuration="22.675851991s" totalImagesPullingTime="17.913559968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:59.630412766 +0000 UTC m=+22.753260273" watchObservedRunningTime="2026-09-16 18:45:59.675851991 +0000 UTC m=+22.798699497" Sep 16 18:45:59.938405 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:45:59.938354 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 18:46:00.342731 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:00.342637 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:59.938369836Z","UUID":"07d3f0c0-4e7f-4be5-b2d1-3e07ac7ae5c0","Handler":null,"Name":"","Endpoint":""} Sep 16 18:46:00.344439 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:00.344417 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 18:46:00.344562 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:00.344449 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 18:46:00.395920 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:00.395892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:00.396304 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:00.395981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:46:00.522462 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:00.522417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" event={"ID":"6c9ee36a-4dc3-4b11-a07a-1a1f18480286","Type":"ContainerStarted","Data":"21e69fbe333c0c163a520f6d687a0dd13c74b1f51e6eb425c9b4c49ad0a4ff16"} Sep 16 18:46:00.524457 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:00.524423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rlpsk" event={"ID":"6c6655a8-ef36-4cd3-a422-1e628e7d4aa4","Type":"ContainerStarted","Data":"287c856c5b193a4e5027707498b8f1a702a2d54885bfdce5cd5a5ffaaf4a4cc8"} Sep 16 18:46:01.329333 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.329307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:46:01.396121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.396096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:01.396449 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:01.396228 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:46:01.396703 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.396679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:01.396821 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:01.396792 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:46:01.529091 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.529014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" event={"ID":"6c9ee36a-4dc3-4b11-a07a-1a1f18480286","Type":"ContainerStarted","Data":"1d50cebc3567798d1ed10924cc44c0995d3f65411cc5d0d129f6a8e324dac760"} Sep 16 18:46:01.533040 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.533011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"f50e3d9d5958a07efffb39d1a26fe0472ceb8550dba9b52ad869949d33164fca"} Sep 16 18:46:01.560710 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.560646 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rlpsk" podStartSLOduration=6.387389679 podStartE2EDuration="24.560627288s" totalImagesPullingTime="18.173237609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:00.542270644 +0000 UTC m=+23.665118151" watchObservedRunningTime="2026-09-16 18:46:01.560627288 +0000 UTC m=+24.683474795" Sep 16 18:46:01.560877 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:01.560851 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5r2t7" podStartSLOduration=4.315845106 podStartE2EDuration="24.560836373s" totalImagesPullingTime="20.244991267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.56006123 +0000 UTC m=+24.682908735" watchObservedRunningTime="2026-09-16 18:46:01.560836373 +0000 UTC m=+24.683683880" Sep 16 18:46:02.395746 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:02.395717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:02.395912 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:02.395837 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:46:02.533050 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:02.533018 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:46:02.533654 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:02.533629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:46:02.535587 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:02.535565 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sgdnp" Sep 16 18:46:03.396022 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:03.395987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:03.396182 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:03.396123 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:46:03.396182 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:03.396153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:03.396285 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:03.396255 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:46:04.396679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:04.396482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:04.397226 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:04.396721 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:46:04.540828 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:04.540799 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cd8ecf3-a922-4d6e-a312-5a775bb163c0" containerID="db8e0de3dbba5402591b53dd03d5364caba61f6b001f6eef82ac023e9bf43cfc" exitCode=0 Sep 16 18:46:04.540967 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:04.540870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerDied","Data":"db8e0de3dbba5402591b53dd03d5364caba61f6b001f6eef82ac023e9bf43cfc"} Sep 16 18:46:04.544680 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:04.544660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" event={"ID":"6f03d949-aa22-4e8b-88d8-db70fd67f5b4","Type":"ContainerStarted","Data":"b55aff24d18c71c27b52dd8120d1c6af1e21555ae8e54b6500271824b7b90cbf"} Sep 16 18:46:05.396284 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.396250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:05.396561 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:05.396406 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:46:05.396561 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.396410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:05.396561 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:05.396480 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:46:05.547990 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.547826 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:46:05.548373 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.548087 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:46:05.548373 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.548192 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:46:05.564889 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.564842 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:46:05.565160 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.565136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:46:05.600931 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.600890 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" podStartSLOduration=10.371219159 podStartE2EDuration="28.600874528s" totalImagesPullingTime="18.229655369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.6297302 +0000 UTC m=+27.752577705" watchObservedRunningTime="2026-09-16 18:46:05.600874528 +0000 UTC m=+28.723722058" Sep 16 18:46:05.715050 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.714947 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jxqsh"] Sep 16 18:46:05.715144 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.715050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:05.715144 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:05.715129 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:46:05.716197 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.716172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t6x4g"] Sep 16 18:46:05.716292 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.716258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:05.716348 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:05.716331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:46:05.716842 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.716817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8tzg8"] Sep 16 18:46:05.716934 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:05.716902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:05.717028 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:05.717011 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:46:06.551812 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:06.551781 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cd8ecf3-a922-4d6e-a312-5a775bb163c0" containerID="2d15dd69ca98f0e399eaa2eb625828e433a3aff63a939c1054d58b8583d0bbed" exitCode=0 Sep 16 18:46:06.552232 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:06.551863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerDied","Data":"2d15dd69ca98f0e399eaa2eb625828e433a3aff63a939c1054d58b8583d0bbed"} Sep 16 18:46:07.396861 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:07.396837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:07.397001 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:07.396940 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:46:07.397001 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:07.396948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:07.397127 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:07.397030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:46:07.397127 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:07.397055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:07.397210 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:07.397136 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:46:07.557150 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:07.557116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerStarted","Data":"9e2da0761d3e6fb89b19104f604d54afc71eba394a53b0d94b83bf889fc7db6f"} Sep 16 18:46:08.564667 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:08.564504 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cd8ecf3-a922-4d6e-a312-5a775bb163c0" containerID="9e2da0761d3e6fb89b19104f604d54afc71eba394a53b0d94b83bf889fc7db6f" exitCode=0 Sep 16 18:46:08.564992 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:08.564569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerDied","Data":"9e2da0761d3e6fb89b19104f604d54afc71eba394a53b0d94b83bf889fc7db6f"} Sep 16 18:46:09.395959 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:09.395928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:09.396118 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:09.395969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:09.396118 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:09.395926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:09.396118 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:09.396065 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:46:09.396249 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:09.396161 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6x4g" podUID="b087b7c6-3e60-4ae6-a9a1-34cd06620b0b" Sep 16 18:46:09.396291 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:09.396252 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxqsh" podUID="51e4870c-c32a-4cc1-b8c8-c0097c2f240c" Sep 16 18:46:11.030126 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.030087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:11.030591 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.030144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:11.030591 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.030217 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:46:11.030591 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.030239 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:46:11.030591 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.030293 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret podName:51e4870c-c32a-4cc1-b8c8-c0097c2f240c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:43.030272056 +0000 UTC m=+66.153119540 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret") pod "global-pull-secret-syncer-jxqsh" (UID: "51e4870c-c32a-4cc1-b8c8-c0097c2f240c") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:46:11.030591 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.030316 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:43.030305545 +0000 UTC m=+66.153153032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:46:11.131015 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.130978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:11.131172 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.131152 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:46:11.131233 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.131180 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:46:11.131233 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.131194 2580 projected.go:196] Error preparing data for projected volume kube-api-access-f8cbz for pod openshift-network-diagnostics/network-check-target-t6x4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:46:11.131323 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.131246 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz podName:b087b7c6-3e60-4ae6-a9a1-34cd06620b0b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:43.131232833 +0000 UTC m=+66.254080329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8cbz" (UniqueName: "kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz") pod "network-check-target-t6x4g" (UID: "b087b7c6-3e60-4ae6-a9a1-34cd06620b0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:46:11.200984 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.200947 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-157.ec2.internal" event="NodeReady" Sep 16 18:46:11.201127 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.201115 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 18:46:11.279682 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.279617 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4sb7x"] Sep 16 18:46:11.284098 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.284070 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bmgfk"] Sep 16 18:46:11.284235 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.284185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.287764 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.287741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.289893 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.289869 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wrlvp\"" Sep 16 18:46:11.290151 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.290131 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 18:46:11.290391 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.290351 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 18:46:11.290484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.290455 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 18:46:11.291107 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.291089 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 18:46:11.291264 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.291091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 18:46:11.291460 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.291444 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nqnnl\"" Sep 16 18:46:11.309216 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.309192 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4sb7x"] Sep 16 18:46:11.309510 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.309456 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bmgfk"] Sep 16 18:46:11.396051 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.396025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:11.396211 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.396028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:11.396497 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.396036 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:11.400297 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.400274 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:46:11.400912 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.400890 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:46:11.400912 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.400908 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:46:11.401151 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.401126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4ccjj\"" Sep 16 18:46:11.401151 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.401130 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:46:11.401748 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.401733 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pl7qs\"" Sep 16 18:46:11.433624 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.433599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1985d5f3-1e2f-46aa-8688-e2491814291f-config-volume\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.433717 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.433633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxk7l\" (UniqueName: \"kubernetes.io/projected/1985d5f3-1e2f-46aa-8688-e2491814291f-kube-api-access-mxk7l\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.433717 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.433705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1985d5f3-1e2f-46aa-8688-e2491814291f-tmp-dir\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.433827 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.433732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7c1f908-5a84-4373-8039-cb6bbab6b949-cert\") pod \"ingress-canary-4sb7x\" (UID: \"c7c1f908-5a84-4373-8039-cb6bbab6b949\") " pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.433827 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.433754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.433926 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.433857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dwv\" (UniqueName: \"kubernetes.io/projected/c7c1f908-5a84-4373-8039-cb6bbab6b949-kube-api-access-58dwv\") pod \"ingress-canary-4sb7x\" (UID: \"c7c1f908-5a84-4373-8039-cb6bbab6b949\") " pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.534433 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dwv\" (UniqueName: \"kubernetes.io/projected/c7c1f908-5a84-4373-8039-cb6bbab6b949-kube-api-access-58dwv\") pod \"ingress-canary-4sb7x\" (UID: \"c7c1f908-5a84-4373-8039-cb6bbab6b949\") " pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.534433 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1985d5f3-1e2f-46aa-8688-e2491814291f-config-volume\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.534598 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk7l\" (UniqueName: \"kubernetes.io/projected/1985d5f3-1e2f-46aa-8688-e2491814291f-kube-api-access-mxk7l\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.534598 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1985d5f3-1e2f-46aa-8688-e2491814291f-tmp-dir\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.534598 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7c1f908-5a84-4373-8039-cb6bbab6b949-cert\") pod \"ingress-canary-4sb7x\" (UID: \"c7c1f908-5a84-4373-8039-cb6bbab6b949\") " pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.534768 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.534820 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.534765 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:11.534869 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.534813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1985d5f3-1e2f-46aa-8688-e2491814291f-tmp-dir\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.534869 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:11.534830 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:12.034811403 +0000 UTC m=+35.157658906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:11.535151 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.535127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1985d5f3-1e2f-46aa-8688-e2491814291f-config-volume\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.539024 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.539002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7c1f908-5a84-4373-8039-cb6bbab6b949-cert\") pod \"ingress-canary-4sb7x\" (UID: \"c7c1f908-5a84-4373-8039-cb6bbab6b949\") " pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.544503 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.544484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dwv\" (UniqueName: \"kubernetes.io/projected/c7c1f908-5a84-4373-8039-cb6bbab6b949-kube-api-access-58dwv\") pod \"ingress-canary-4sb7x\" (UID: \"c7c1f908-5a84-4373-8039-cb6bbab6b949\") " pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:11.544755 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.544736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk7l\" (UniqueName: \"kubernetes.io/projected/1985d5f3-1e2f-46aa-8688-e2491814291f-kube-api-access-mxk7l\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:11.597346 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:11.597321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4sb7x" Sep 16 18:46:12.037735 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:12.037696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:12.038359 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:12.037851 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:12.038359 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:12.037929 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.037907554 +0000 UTC m=+36.160755058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:12.731519 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:12.731233 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4sb7x"] Sep 16 18:46:12.752324 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:46:12.752279 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c1f908_5a84_4373_8039_cb6bbab6b949.slice/crio-aed10715d44e2175e9281dc9f6a1b01ff739f65a2d615a84f4a180a561971a26 WatchSource:0}: Error finding container aed10715d44e2175e9281dc9f6a1b01ff739f65a2d615a84f4a180a561971a26: Status 404 returned error can't find the container with id aed10715d44e2175e9281dc9f6a1b01ff739f65a2d615a84f4a180a561971a26 Sep 16 18:46:13.044936 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:13.044905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:13.045374 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:13.045028 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:13.045374 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:13.045082 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.045067221 +0000 UTC m=+38.167914705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:13.580803 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:13.580769 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cd8ecf3-a922-4d6e-a312-5a775bb163c0" containerID="627c973b77eea849e6264187c7d9058ff6edad9f1a7e83d2f41633e9ff3d9958" exitCode=0 Sep 16 18:46:13.580963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:13.580851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerDied","Data":"627c973b77eea849e6264187c7d9058ff6edad9f1a7e83d2f41633e9ff3d9958"} Sep 16 18:46:13.582193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:13.582164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4sb7x" event={"ID":"c7c1f908-5a84-4373-8039-cb6bbab6b949","Type":"ContainerStarted","Data":"aed10715d44e2175e9281dc9f6a1b01ff739f65a2d615a84f4a180a561971a26"} Sep 16 18:46:14.588581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:14.588548 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cd8ecf3-a922-4d6e-a312-5a775bb163c0" containerID="bb80b466c77ee937157995cb69c9b40d9aaec33506faa34d1100ad1ab6d9bb55" exitCode=0 Sep 16 18:46:14.589026 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:14.588590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerDied","Data":"bb80b466c77ee937157995cb69c9b40d9aaec33506faa34d1100ad1ab6d9bb55"} Sep 16 18:46:15.061459 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:15.061432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:15.061604 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:15.061584 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:15.061719 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:15.061657 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.061635686 +0000 UTC m=+42.184483184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:15.592919 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:15.592709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4sb7x" event={"ID":"c7c1f908-5a84-4373-8039-cb6bbab6b949","Type":"ContainerStarted","Data":"e3361558fb63f88901d699d316ebf7a68fe5a57a02d1d584314512786cc93088"} Sep 16 18:46:15.596498 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:15.596474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pg56t" event={"ID":"4cd8ecf3-a922-4d6e-a312-5a775bb163c0","Type":"ContainerStarted","Data":"a156c241c59ac8e400120f044c8f10a9066c4b178a783b76cc13392095b51470"} Sep 16 18:46:15.612760 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:15.612711 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4sb7x" podStartSLOduration=2.1604176920000002 podStartE2EDuration="4.612682541s" totalImagesPullingTime="2.452264849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:15.612479897 +0000 UTC m=+38.735327404" watchObservedRunningTime="2026-09-16 18:46:15.612682541 +0000 UTC m=+38.735530077" Sep 16 18:46:15.638586 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:15.638521 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pg56t" podStartSLOduration=9.755557133 podStartE2EDuration="38.638501103s" totalImagesPullingTime="28.88294397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:15.636710535 +0000 UTC m=+38.759558070" watchObservedRunningTime="2026-09-16 18:46:15.638501103 +0000 UTC m=+38.761348612" Sep 16 18:46:19.089356 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:19.089310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:19.089791 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:19.089475 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:19.089791 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:19.089551 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:27.089532229 +0000 UTC m=+50.212379732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:27.145632 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:27.145599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:27.146044 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:27.145740 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:27.146044 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:27.145811 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:43.145795686 +0000 UTC m=+66.268643170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:37.567170 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:37.567143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gz585" Sep 16 18:46:43.049640 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.049599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:43.050125 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.049650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:46:43.052147 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.052117 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:46:43.052266 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.052162 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:46:43.060836 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:43.060815 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:46:43.060932 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:43.060873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:47.060859824 +0000 UTC m=+130.183707308 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : secret "metrics-daemon-secret" not found Sep 16 18:46:43.064372 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.064349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51e4870c-c32a-4cc1-b8c8-c0097c2f240c-original-pull-secret\") pod \"global-pull-secret-syncer-jxqsh\" (UID: \"51e4870c-c32a-4cc1-b8c8-c0097c2f240c\") " pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:43.150138 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.150112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:46:43.150231 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.150158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:43.150270 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:43.150249 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:43.150304 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:46:43.150296 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.150282378 +0000 UTC m=+98.273129877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:46:43.152858 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.152839 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:46:43.163071 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.163049 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:46:43.174155 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.174137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8cbz\" (UniqueName: \"kubernetes.io/projected/b087b7c6-3e60-4ae6-a9a1-34cd06620b0b-kube-api-access-f8cbz\") pod \"network-check-target-t6x4g\" (UID: \"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b\") " pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:43.209862 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.209840 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4ccjj\"" Sep 16 18:46:43.213949 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.213935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxqsh" Sep 16 18:46:43.217997 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.217982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:43.513723 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.513687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jxqsh"] Sep 16 18:46:43.517064 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:46:43.517042 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e4870c_c32a_4cc1_b8c8_c0097c2f240c.slice/crio-60a89e34b5540ef355b180ea265eafc62747f4be5fa598ec6d3ed0ff0c90e69b WatchSource:0}: Error finding container 60a89e34b5540ef355b180ea265eafc62747f4be5fa598ec6d3ed0ff0c90e69b: Status 404 returned error can't find the container with id 60a89e34b5540ef355b180ea265eafc62747f4be5fa598ec6d3ed0ff0c90e69b Sep 16 18:46:43.541907 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.541887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t6x4g"] Sep 16 18:46:43.544280 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:46:43.544261 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb087b7c6_3e60_4ae6_a9a1_34cd06620b0b.slice/crio-e56f09272c43835efa50fcce979707a8f04629deb86b53fc0b3590affe1b7c29 WatchSource:0}: Error finding container e56f09272c43835efa50fcce979707a8f04629deb86b53fc0b3590affe1b7c29: Status 404 returned error can't find the container with id e56f09272c43835efa50fcce979707a8f04629deb86b53fc0b3590affe1b7c29 Sep 16 18:46:43.660450 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.660360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t6x4g" event={"ID":"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b","Type":"ContainerStarted","Data":"e56f09272c43835efa50fcce979707a8f04629deb86b53fc0b3590affe1b7c29"} Sep 16 18:46:43.661500 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:43.661478 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jxqsh" event={"ID":"51e4870c-c32a-4cc1-b8c8-c0097c2f240c","Type":"ContainerStarted","Data":"60a89e34b5540ef355b180ea265eafc62747f4be5fa598ec6d3ed0ff0c90e69b"} Sep 16 18:46:48.680702 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:48.680664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t6x4g" event={"ID":"b087b7c6-3e60-4ae6-a9a1-34cd06620b0b","Type":"ContainerStarted","Data":"d1d72279c7a70f5e6f66ddfe871d9bedc7db6ad574331160ac862d8bb5aea76b"} Sep 16 18:46:48.681218 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:48.680778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:46:48.682209 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:48.682188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jxqsh" event={"ID":"51e4870c-c32a-4cc1-b8c8-c0097c2f240c","Type":"ContainerStarted","Data":"fcba2fa74212cf2bb46902b53284ec032bc3641f9594df94847bd2265ae9672a"} Sep 16 18:46:48.699117 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:48.699077 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t6x4g" podStartSLOduration=67.255362769 podStartE2EDuration="1m11.699065692s" totalImagesPullingTime="4.443702923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:48.698148009 +0000 UTC m=+71.820995516" watchObservedRunningTime="2026-09-16 18:46:48.699065692 +0000 UTC m=+71.821913176" Sep 16 18:46:48.717802 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:46:48.717759 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jxqsh" podStartSLOduration=67.244263531 podStartE2EDuration="1m11.717745152s" totalImagesPullingTime="4.473481621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:48.71765795 +0000 UTC m=+71.840505457" watchObservedRunningTime="2026-09-16 18:46:48.717745152 +0000 UTC m=+71.840592659" Sep 16 18:47:15.246687 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:15.246580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:47:15.247121 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:15.246691 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:47:15.247121 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:15.246742 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls podName:1985d5f3-1e2f-46aa-8688-e2491814291f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:19.246730033 +0000 UTC m=+162.369577529 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls") pod "dns-default-bmgfk" (UID: "1985d5f3-1e2f-46aa-8688-e2491814291f") : secret "dns-default-metrics-tls" not found Sep 16 18:47:19.686461 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:19.686430 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t6x4g" Sep 16 18:47:38.032963 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.032918 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-qdlfp"] Sep 16 18:47:38.035525 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.035495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.042914 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.042895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2297t\"" Sep 16 18:47:38.043840 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.043827 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 18:47:38.043964 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.043935 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 18:47:38.045161 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.045147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 18:47:38.045609 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.045596 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 18:47:38.049506 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.049492 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 18:47:38.055364 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.055342 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qdlfp"] Sep 16 18:47:38.076205 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.076178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-service-ca-bundle\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.076302 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.076213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-snapshots\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.076302 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.076237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.076302 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.076277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-tmp\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.076491 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.076337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-serving-cert\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.076491 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.076423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-kube-api-access-4w872\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.176970 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.176948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-serving-cert\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177053 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.176989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-kube-api-access-4w872\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177053 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-service-ca-bundle\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177053 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-snapshots\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177053 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177214 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-tmp\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177631 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-service-ca-bundle\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177631 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-tmp\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.177772 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.177651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-snapshots\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.178078 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.178056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.179532 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.179516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-serving-cert\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.188472 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.188443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/f9a248c6-76bc-457d-a2b8-c4b5231a04bb-kube-api-access-4w872\") pod \"insights-operator-5844b856db-qdlfp\" (UID: \"f9a248c6-76bc-457d-a2b8-c4b5231a04bb\") " pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.343449 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.343427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qdlfp" Sep 16 18:47:38.614033 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.613963 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qdlfp"] Sep 16 18:47:38.617750 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:47:38.617726 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a248c6_76bc_457d_a2b8_c4b5231a04bb.slice/crio-5b5283d83144661e90ff2b6ceb3bedadce8597b742b34ecc4234a30052891d84 WatchSource:0}: Error finding container 5b5283d83144661e90ff2b6ceb3bedadce8597b742b34ecc4234a30052891d84: Status 404 returned error can't find the container with id 5b5283d83144661e90ff2b6ceb3bedadce8597b742b34ecc4234a30052891d84 Sep 16 18:47:38.792006 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:38.791976 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qdlfp" event={"ID":"f9a248c6-76bc-457d-a2b8-c4b5231a04bb","Type":"ContainerStarted","Data":"5b5283d83144661e90ff2b6ceb3bedadce8597b742b34ecc4234a30052891d84"} Sep 16 18:47:40.799552 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:40.799515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qdlfp" event={"ID":"f9a248c6-76bc-457d-a2b8-c4b5231a04bb","Type":"ContainerStarted","Data":"f01aef75dd658409d9afd6b2ffc01198edc431ffc5af8082cecc25129704014a"} Sep 16 18:47:40.822559 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:40.822508 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-qdlfp" podStartSLOduration=2.166977657 podStartE2EDuration="3.82249322s" totalImagesPullingTime="1.655515563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:40.822211889 +0000 UTC m=+123.945059396" watchObservedRunningTime="2026-09-16 18:47:40.82249322 +0000 UTC m=+123.945340725" Sep 16 18:47:41.806608 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.806576 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6wdmf"] Sep 16 18:47:41.808411 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.808397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:41.813086 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.813059 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 18:47:41.813086 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.813068 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 18:47:41.816082 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.816066 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kb6xm\"" Sep 16 18:47:41.842263 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.842241 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6wdmf"] Sep 16 18:47:41.904128 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.904101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:41.904214 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.904140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f2ac848-9cd2-42f2-97ae-74f25458627b-crio-socket\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:41.904214 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.904181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mrj\" (UniqueName: \"kubernetes.io/projected/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-api-access-27mrj\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:41.904325 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.904308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f2ac848-9cd2-42f2-97ae-74f25458627b-data-volume\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:41.904364 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:41.904353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005401 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005481 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f2ac848-9cd2-42f2-97ae-74f25458627b-crio-socket\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005481 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mrj\" (UniqueName: \"kubernetes.io/projected/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-api-access-27mrj\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005590 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f2ac848-9cd2-42f2-97ae-74f25458627b-data-volume\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005590 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.005500 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:42.005590 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005590 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.005569 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.505549918 +0000 UTC m=+125.628397419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:42.005590 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f2ac848-9cd2-42f2-97ae-74f25458627b-crio-socket\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.005590 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.005592 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:42.005851 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.005645 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.505636333 +0000 UTC m=+125.628483817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : configmap "kube-rbac-proxy" not found Sep 16 18:47:42.005851 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.005799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f2ac848-9cd2-42f2-97ae-74f25458627b-data-volume\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.018324 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.018305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mrj\" (UniqueName: \"kubernetes.io/projected/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-api-access-27mrj\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.509363 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.509338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.509489 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:42.509392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:42.509489 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.509443 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:42.509489 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.509473 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:42.509602 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.509492 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.509479976 +0000 UTC m=+126.632327499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : configmap "kube-rbac-proxy" not found Sep 16 18:47:42.509602 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:42.509515 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.50950543 +0000 UTC m=+126.632352915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:43.516727 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:43.516702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:43.517185 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:43.516739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:43.517185 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:43.516831 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:43.517185 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:43.516846 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:43.517185 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:43.516892 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.516879737 +0000 UTC m=+128.639727234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:43.517185 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:43.516903 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.516897639 +0000 UTC m=+128.639745122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : configmap "kube-rbac-proxy" not found Sep 16 18:47:43.613083 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:43.613064 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vshfx_e50d547c-882d-4029-96ab-6d960ba6b3c1/dns-node-resolver/0.log" Sep 16 18:47:44.214616 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:44.214592 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dw5h8_3d752bc5-c177-4081-8396-b249b8eacd3f/node-ca/0.log" Sep 16 18:47:45.014666 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.014641 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4sb7x_c7c1f908-5a84-4373-8039-cb6bbab6b949/serve-healthcheck-canary/0.log" Sep 16 18:47:45.529090 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.529057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:45.529285 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.529097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:45.529285 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:45.529172 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:45.529285 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:45.529204 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:45.529285 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:45.529245 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.529230167 +0000 UTC m=+132.652077651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : configmap "kube-rbac-proxy" not found Sep 16 18:47:45.529285 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:45.529260 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.529253994 +0000 UTC m=+132.652101478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:45.973630 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.973600 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr"] Sep 16 18:47:45.975557 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.975542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:45.979703 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.979680 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6896b\"" Sep 16 18:47:45.979801 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.979717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 18:47:45.980416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.980371 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:45.980527 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.980418 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:45.996519 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:45.996501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr"] Sep 16 18:47:46.031725 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.031702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:46.032005 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.031770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kst\" (UniqueName: \"kubernetes.io/projected/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-kube-api-access-d2kst\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:46.072327 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.072300 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl"] Sep 16 18:47:46.073991 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.073975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" Sep 16 18:47:46.077647 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.077630 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:46.078851 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.078834 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:46.078942 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.078895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xr7jg\"" Sep 16 18:47:46.095484 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.095459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl"] Sep 16 18:47:46.132267 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.132242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:46.132351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.132315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kst\" (UniqueName: \"kubernetes.io/projected/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-kube-api-access-d2kst\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:46.132351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.132345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjv2\" (UniqueName: \"kubernetes.io/projected/4ef3f29c-6555-48e5-975f-9236bda75973-kube-api-access-ghjv2\") pod \"volume-data-source-validator-86695b7cc7-gcrnl\" (UID: \"4ef3f29c-6555-48e5-975f-9236bda75973\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" Sep 16 18:47:46.132455 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:46.132411 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:46.132497 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:46.132478 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls podName:3b1fcd05-fcfa-4305-a1e1-5aa04958ad77 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:46.632460184 +0000 UTC m=+129.755307682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-p2nlr" (UID: "3b1fcd05-fcfa-4305-a1e1-5aa04958ad77") : secret "samples-operator-tls" not found Sep 16 18:47:46.151444 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.151417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kst\" (UniqueName: \"kubernetes.io/projected/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-kube-api-access-d2kst\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:46.233605 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.233557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjv2\" (UniqueName: \"kubernetes.io/projected/4ef3f29c-6555-48e5-975f-9236bda75973-kube-api-access-ghjv2\") pod \"volume-data-source-validator-86695b7cc7-gcrnl\" (UID: \"4ef3f29c-6555-48e5-975f-9236bda75973\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" Sep 16 18:47:46.245758 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.245733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjv2\" (UniqueName: \"kubernetes.io/projected/4ef3f29c-6555-48e5-975f-9236bda75973-kube-api-access-ghjv2\") pod \"volume-data-source-validator-86695b7cc7-gcrnl\" (UID: \"4ef3f29c-6555-48e5-975f-9236bda75973\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" Sep 16 18:47:46.382447 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.382422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" Sep 16 18:47:46.635496 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.635471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:46.635634 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:46.635609 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:46.635691 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:46.635680 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls podName:3b1fcd05-fcfa-4305-a1e1-5aa04958ad77 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:47.635664967 +0000 UTC m=+130.758512450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-p2nlr" (UID: "3b1fcd05-fcfa-4305-a1e1-5aa04958ad77") : secret "samples-operator-tls" not found Sep 16 18:47:46.658128 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.658104 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl"] Sep 16 18:47:46.661057 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:47:46.661036 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef3f29c_6555_48e5_975f_9236bda75973.slice/crio-0948784fc10d96babb7990dd6e3f01812844f98ed3252b078910def50bf6af9a WatchSource:0}: Error finding container 0948784fc10d96babb7990dd6e3f01812844f98ed3252b078910def50bf6af9a: Status 404 returned error can't find the container with id 0948784fc10d96babb7990dd6e3f01812844f98ed3252b078910def50bf6af9a Sep 16 18:47:46.814085 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.814064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" event={"ID":"4ef3f29c-6555-48e5-975f-9236bda75973","Type":"ContainerStarted","Data":"0948784fc10d96babb7990dd6e3f01812844f98ed3252b078910def50bf6af9a"} Sep 16 18:47:46.993272 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.993217 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-ls8jm"] Sep 16 18:47:46.995761 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:46.995746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.000805 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.000787 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 18:47:47.001190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.001170 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:47.001279 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.001190 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 18:47:47.001647 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.001632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kkzd8\"" Sep 16 18:47:47.001999 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.001985 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:47.007842 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.007826 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 18:47:47.018199 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.018179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-ls8jm"] Sep 16 18:47:47.037902 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.037875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cf7deaa-e85f-4eda-980b-04cb0d77d071-config\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.038173 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.037913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsx92\" (UniqueName: \"kubernetes.io/projected/7cf7deaa-e85f-4eda-980b-04cb0d77d071-kube-api-access-fsx92\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.038173 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.037974 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cf7deaa-e85f-4eda-980b-04cb0d77d071-serving-cert\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.038173 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.037995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cf7deaa-e85f-4eda-980b-04cb0d77d071-trusted-ca\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.138485 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.138459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:47:47.138581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.138513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cf7deaa-e85f-4eda-980b-04cb0d77d071-serving-cert\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.138581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.138538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cf7deaa-e85f-4eda-980b-04cb0d77d071-trusted-ca\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.138684 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:47.138613 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:47:47.138684 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.138622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cf7deaa-e85f-4eda-980b-04cb0d77d071-config\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.138684 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.138647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsx92\" (UniqueName: \"kubernetes.io/projected/7cf7deaa-e85f-4eda-980b-04cb0d77d071-kube-api-access-fsx92\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.138827 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:47.138687 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs podName:593cdcf1-d960-4800-a58f-46e20476cb32 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:49.138669294 +0000 UTC m=+252.261516777 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs") pod "network-metrics-daemon-8tzg8" (UID: "593cdcf1-d960-4800-a58f-46e20476cb32") : secret "metrics-daemon-secret" not found Sep 16 18:47:47.139569 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.139546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cf7deaa-e85f-4eda-980b-04cb0d77d071-config\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.139710 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.139666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cf7deaa-e85f-4eda-980b-04cb0d77d071-trusted-ca\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.141249 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.141227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cf7deaa-e85f-4eda-980b-04cb0d77d071-serving-cert\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.148834 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.148811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsx92\" (UniqueName: \"kubernetes.io/projected/7cf7deaa-e85f-4eda-980b-04cb0d77d071-kube-api-access-fsx92\") pod \"console-operator-696884fb9c-ls8jm\" (UID: \"7cf7deaa-e85f-4eda-980b-04cb0d77d071\") " pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.304696 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.304673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:47.585195 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.583985 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-ls8jm"] Sep 16 18:47:47.588748 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:47:47.588723 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf7deaa_e85f_4eda_980b_04cb0d77d071.slice/crio-fafb4be7fd439c0c22e867f2f5c2169f689e19b768a695650f747ec11c7ba6ab WatchSource:0}: Error finding container fafb4be7fd439c0c22e867f2f5c2169f689e19b768a695650f747ec11c7ba6ab: Status 404 returned error can't find the container with id fafb4be7fd439c0c22e867f2f5c2169f689e19b768a695650f747ec11c7ba6ab Sep 16 18:47:47.642111 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.642088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:47.642229 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:47.642212 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:47.642270 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:47.642264 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls podName:3b1fcd05-fcfa-4305-a1e1-5aa04958ad77 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.642251404 +0000 UTC m=+132.765098887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-p2nlr" (UID: "3b1fcd05-fcfa-4305-a1e1-5aa04958ad77") : secret "samples-operator-tls" not found Sep 16 18:47:47.817861 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.817833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" event={"ID":"4ef3f29c-6555-48e5-975f-9236bda75973","Type":"ContainerStarted","Data":"19858ec826ce265ce0bef6cebe7c1934a42a5887770e2cd40e7b3292d6ec1921"} Sep 16 18:47:47.818951 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.818929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" event={"ID":"7cf7deaa-e85f-4eda-980b-04cb0d77d071","Type":"ContainerStarted","Data":"fafb4be7fd439c0c22e867f2f5c2169f689e19b768a695650f747ec11c7ba6ab"} Sep 16 18:47:47.848675 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.848609 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcrnl" podStartSLOduration=0.890475813 podStartE2EDuration="1.848594655s" totalImagesPullingTime="958.118842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:47.848062683 +0000 UTC m=+130.970910187" watchObservedRunningTime="2026-09-16 18:47:47.848594655 +0000 UTC m=+130.971442158" Sep 16 18:47:47.973672 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.973648 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x"] Sep 16 18:47:47.976420 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.976405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" Sep 16 18:47:47.979235 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:47.979215 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hzdgl\"" Sep 16 18:47:48.030329 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.030301 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x"] Sep 16 18:47:48.046476 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.046455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fttl\" (UniqueName: \"kubernetes.io/projected/edbdd95d-89d8-4559-aca5-cb5ba13283f9-kube-api-access-6fttl\") pod \"network-check-source-6f8b768455-tfj8x\" (UID: \"edbdd95d-89d8-4559-aca5-cb5ba13283f9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" Sep 16 18:47:48.147873 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.147820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fttl\" (UniqueName: \"kubernetes.io/projected/edbdd95d-89d8-4559-aca5-cb5ba13283f9-kube-api-access-6fttl\") pod \"network-check-source-6f8b768455-tfj8x\" (UID: \"edbdd95d-89d8-4559-aca5-cb5ba13283f9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" Sep 16 18:47:48.161497 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.161474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fttl\" (UniqueName: \"kubernetes.io/projected/edbdd95d-89d8-4559-aca5-cb5ba13283f9-kube-api-access-6fttl\") pod \"network-check-source-6f8b768455-tfj8x\" (UID: \"edbdd95d-89d8-4559-aca5-cb5ba13283f9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" Sep 16 18:47:48.284303 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.284279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" Sep 16 18:47:48.562456 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.562427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x"] Sep 16 18:47:48.564704 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:47:48.564618 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbdd95d_89d8_4559_aca5_cb5ba13283f9.slice/crio-70624fe6fb3ef091d278c5dd0d35a5362e2cf1002a42a8d1274eaa6c4f9291cb WatchSource:0}: Error finding container 70624fe6fb3ef091d278c5dd0d35a5362e2cf1002a42a8d1274eaa6c4f9291cb: Status 404 returned error can't find the container with id 70624fe6fb3ef091d278c5dd0d35a5362e2cf1002a42a8d1274eaa6c4f9291cb Sep 16 18:47:48.824834 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.824755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" event={"ID":"edbdd95d-89d8-4559-aca5-cb5ba13283f9","Type":"ContainerStarted","Data":"a2b3dea3085ed96740c81975f9ec142ec50cd8501e1a7320b8c735263140c090"} Sep 16 18:47:48.824834 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.824822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" event={"ID":"edbdd95d-89d8-4559-aca5-cb5ba13283f9","Type":"ContainerStarted","Data":"70624fe6fb3ef091d278c5dd0d35a5362e2cf1002a42a8d1274eaa6c4f9291cb"} Sep 16 18:47:48.844900 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:48.844845 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tfj8x" podStartSLOduration=1.8448296320000002 podStartE2EDuration="1.844829632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:48.84432898 +0000 UTC m=+131.967176485" watchObservedRunningTime="2026-09-16 18:47:48.844829632 +0000 UTC m=+131.967677136" Sep 16 18:47:49.558364 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.558339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:49.558701 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.558399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:49.558701 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:49.558463 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:49.558701 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:49.558524 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:57.558508449 +0000 UTC m=+140.681355932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : configmap "kube-rbac-proxy" not found Sep 16 18:47:49.558701 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:49.558551 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:49.558701 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:49.558619 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:57.558602907 +0000 UTC m=+140.681450408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:49.658753 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.658710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:49.658910 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:49.658858 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:49.658958 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:49.658925 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls podName:3b1fcd05-fcfa-4305-a1e1-5aa04958ad77 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.658909214 +0000 UTC m=+136.781756697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-p2nlr" (UID: "3b1fcd05-fcfa-4305-a1e1-5aa04958ad77") : secret "samples-operator-tls" not found Sep 16 18:47:49.828950 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.828923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/0.log" Sep 16 18:47:49.829088 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.828964 2580 generic.go:356] "Generic (PLEG): container finished" podID="7cf7deaa-e85f-4eda-980b-04cb0d77d071" containerID="e60efafd032735e6cd9f87e2c299a8846d45ac4f51a2fd8cf8776486542cedeb" exitCode=255 Sep 16 18:47:49.829088 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.829052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" event={"ID":"7cf7deaa-e85f-4eda-980b-04cb0d77d071","Type":"ContainerDied","Data":"e60efafd032735e6cd9f87e2c299a8846d45ac4f51a2fd8cf8776486542cedeb"} Sep 16 18:47:49.829196 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:49.829183 2580 scope.go:117] "RemoveContainer" containerID="e60efafd032735e6cd9f87e2c299a8846d45ac4f51a2fd8cf8776486542cedeb" Sep 16 18:47:50.833319 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:50.833298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:47:50.833702 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:50.833688 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/0.log" Sep 16 18:47:50.833745 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:50.833721 2580 generic.go:356] "Generic (PLEG): container finished" podID="7cf7deaa-e85f-4eda-980b-04cb0d77d071" containerID="d257970250b7dfa6159de6062f0ebd00c1c76deccdaf2d2275ed377fd757be96" exitCode=255 Sep 16 18:47:50.833779 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:50.833762 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" event={"ID":"7cf7deaa-e85f-4eda-980b-04cb0d77d071","Type":"ContainerDied","Data":"d257970250b7dfa6159de6062f0ebd00c1c76deccdaf2d2275ed377fd757be96"} Sep 16 18:47:50.833832 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:50.833822 2580 scope.go:117] "RemoveContainer" containerID="e60efafd032735e6cd9f87e2c299a8846d45ac4f51a2fd8cf8776486542cedeb" Sep 16 18:47:50.833946 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:50.833933 2580 scope.go:117] "RemoveContainer" containerID="d257970250b7dfa6159de6062f0ebd00c1c76deccdaf2d2275ed377fd757be96" Sep 16 18:47:50.834092 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:50.834071 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-ls8jm_openshift-console-operator(7cf7deaa-e85f-4eda-980b-04cb0d77d071)\"" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" podUID="7cf7deaa-e85f-4eda-980b-04cb0d77d071" Sep 16 18:47:51.838534 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:51.838509 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:47:51.838859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:51.838770 2580 scope.go:117] "RemoveContainer" containerID="d257970250b7dfa6159de6062f0ebd00c1c76deccdaf2d2275ed377fd757be96" Sep 16 18:47:51.838920 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:51.838902 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-ls8jm_openshift-console-operator(7cf7deaa-e85f-4eda-980b-04cb0d77d071)\"" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" podUID="7cf7deaa-e85f-4eda-980b-04cb0d77d071" Sep 16 18:47:53.686082 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:53.686043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:47:53.686457 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:53.686151 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:53.686457 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:53.686202 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls podName:3b1fcd05-fcfa-4305-a1e1-5aa04958ad77 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:01.686189043 +0000 UTC m=+144.809036526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-p2nlr" (UID: "3b1fcd05-fcfa-4305-a1e1-5aa04958ad77") : secret "samples-operator-tls" not found Sep 16 18:47:57.305565 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:57.305536 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:57.305914 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:57.305616 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:47:57.305914 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:57.305900 2580 scope.go:117] "RemoveContainer" containerID="d257970250b7dfa6159de6062f0ebd00c1c76deccdaf2d2275ed377fd757be96" Sep 16 18:47:57.306057 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:57.306038 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-ls8jm_openshift-console-operator(7cf7deaa-e85f-4eda-980b-04cb0d77d071)\"" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" podUID="7cf7deaa-e85f-4eda-980b-04cb0d77d071" Sep 16 18:47:57.610019 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:57.609943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:57.610019 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:57.609992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:47:57.610210 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:57.610085 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:57.610210 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:47:57.610147 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls podName:9f2ac848-9cd2-42f2-97ae-74f25458627b nodeName:}" failed. No retries permitted until 2026-09-16 18:48:13.610130788 +0000 UTC m=+156.732978275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wdmf" (UID: "9f2ac848-9cd2-42f2-97ae-74f25458627b") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:57.610417 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:47:57.610371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f2ac848-9cd2-42f2-97ae-74f25458627b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:48:01.736070 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:01.736035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:48:01.738411 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:01.738393 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b1fcd05-fcfa-4305-a1e1-5aa04958ad77-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-p2nlr\" (UID: \"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:48:01.883681 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:01.883657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" Sep 16 18:48:02.159502 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:02.159476 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr"] Sep 16 18:48:02.873280 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:02.873232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" event={"ID":"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77","Type":"ContainerStarted","Data":"6157a9938b542e50f2344857988bec79f19a3a3b9dae9bfd89a86804accbeb76"} Sep 16 18:48:04.880137 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:04.880100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" event={"ID":"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77","Type":"ContainerStarted","Data":"d714a10699675f85ee5c5867f77cfaf0d59052bbb55764124e588f8ecdbe07ef"} Sep 16 18:48:04.880550 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:04.880153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" event={"ID":"3b1fcd05-fcfa-4305-a1e1-5aa04958ad77","Type":"ContainerStarted","Data":"e8eb019281e3ff51fd9b9cf9408cbb0b1346544276e79b8ac2615ad9de015547"} Sep 16 18:48:04.901954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:04.901911 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-p2nlr" podStartSLOduration=18.230272006 podStartE2EDuration="19.901897008s" totalImagesPullingTime="1.671625002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:04.901540667 +0000 UTC m=+148.024388171" watchObservedRunningTime="2026-09-16 18:48:04.901897008 +0000 UTC m=+148.024744515" Sep 16 18:48:08.396612 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:08.396587 2580 scope.go:117] "RemoveContainer" containerID="d257970250b7dfa6159de6062f0ebd00c1c76deccdaf2d2275ed377fd757be96" Sep 16 18:48:08.893349 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:08.893326 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:48:08.893583 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:08.893445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" event={"ID":"7cf7deaa-e85f-4eda-980b-04cb0d77d071","Type":"ContainerStarted","Data":"1b481261fdb9bb60436f6f2bf720c19d886a3a29a4d92bcd6dc3cdb0d887644e"} Sep 16 18:48:08.893757 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:08.893737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:48:08.920711 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:08.920672 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" podStartSLOduration=21.096453225 podStartE2EDuration="22.92065935s" totalImagesPullingTime="1.824206125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:08.918638531 +0000 UTC m=+152.041486057" watchObservedRunningTime="2026-09-16 18:48:08.92065935 +0000 UTC m=+152.043506856" Sep 16 18:48:09.403240 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:09.403214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-ls8jm" Sep 16 18:48:13.619174 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:13.619141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:48:13.621545 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:13.621523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f2ac848-9cd2-42f2-97ae-74f25458627b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wdmf\" (UID: \"9f2ac848-9cd2-42f2-97ae-74f25458627b\") " pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:48:13.917511 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:13.917452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6wdmf" Sep 16 18:48:14.190663 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:14.190598 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6wdmf"] Sep 16 18:48:14.193438 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:14.193412 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2ac848_9cd2_42f2_97ae_74f25458627b.slice/crio-e854aaa65919daeac5d75e9b171a6d9d364cd60058c2e5760ad123faadeeb4e1 WatchSource:0}: Error finding container e854aaa65919daeac5d75e9b171a6d9d364cd60058c2e5760ad123faadeeb4e1: Status 404 returned error can't find the container with id e854aaa65919daeac5d75e9b171a6d9d364cd60058c2e5760ad123faadeeb4e1 Sep 16 18:48:14.304646 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:48:14.304613 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bmgfk" podUID="1985d5f3-1e2f-46aa-8688-e2491814291f" Sep 16 18:48:14.419798 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:48:14.419773 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8tzg8" podUID="593cdcf1-d960-4800-a58f-46e20476cb32" Sep 16 18:48:14.913778 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:14.913743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wdmf" event={"ID":"9f2ac848-9cd2-42f2-97ae-74f25458627b","Type":"ContainerStarted","Data":"308d7d5901756f1527a3e359edd7df016dd17c931eee384af2eb2801c1faacde"} Sep 16 18:48:14.913778 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:14.913763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmgfk" Sep 16 18:48:14.914155 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:14.913849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wdmf" event={"ID":"9f2ac848-9cd2-42f2-97ae-74f25458627b","Type":"ContainerStarted","Data":"ae36c888fe6155019c78a77dc376ca747fdb65e9aaf2c239eae4504edadab495"} Sep 16 18:48:14.914155 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:14.913886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wdmf" event={"ID":"9f2ac848-9cd2-42f2-97ae-74f25458627b","Type":"ContainerStarted","Data":"e854aaa65919daeac5d75e9b171a6d9d364cd60058c2e5760ad123faadeeb4e1"} Sep 16 18:48:15.812947 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.812766 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7ff954774d-drwgj"] Sep 16 18:48:15.816280 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.816259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.819455 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.819425 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 18:48:15.819574 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.819428 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 18:48:15.820968 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.820950 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5jk86\"" Sep 16 18:48:15.827878 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.827562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 18:48:15.829644 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.829625 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 18:48:15.838667 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.838646 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk"] Sep 16 18:48:15.841795 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.841777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:15.850599 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.850580 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 18:48:15.851152 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.850840 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 18:48:15.851152 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.851003 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 18:48:15.851152 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.851060 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 18:48:15.851350 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.851328 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vkllx\"" Sep 16 18:48:15.873605 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.873549 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk"] Sep 16 18:48:15.874540 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.874505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff954774d-drwgj"] Sep 16 18:48:15.883858 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.883834 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h"] Sep 16 18:48:15.887412 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.887370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:15.889517 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.889483 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-p9pct\"" Sep 16 18:48:15.889628 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.889545 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 18:48:15.889628 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.889600 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 18:48:15.903761 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.903712 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h"] Sep 16 18:48:15.938005 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.937983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-bound-sa-token\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-ca-trust-extracted\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-installation-pull-secrets\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47ceefb6-ebc1-4457-80b8-379f5182242f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77dff77b-mp9sk\" (UID: \"47ceefb6-ebc1-4457-80b8-379f5182242f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-certificates\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwfw\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-kube-api-access-8rwfw\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-trusted-ca\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-tls\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938353 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-image-registry-private-configuration\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:15.938717 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.938366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkp2\" (UniqueName: \"kubernetes.io/projected/47ceefb6-ebc1-4457-80b8-379f5182242f-kube-api-access-vqkp2\") pod \"managed-serviceaccount-addon-agent-77dff77b-mp9sk\" (UID: \"47ceefb6-ebc1-4457-80b8-379f5182242f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:15.948788 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.948764 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ff954774d-drwgj"] Sep 16 18:48:15.948950 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:48:15.948931 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-8rwfw registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7ff954774d-drwgj" podUID="a7f46fa1-1fab-46ef-b4be-a28cba8233e0" Sep 16 18:48:15.980813 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.980793 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8"] Sep 16 18:48:15.983817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.983800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:15.985923 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.985895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lv54z\"" Sep 16 18:48:15.986438 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:15.986419 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 18:48:16.031102 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.031069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-667c9bc59c-n25ng"] Sep 16 18:48:16.035634 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.035612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.038713 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-certificates\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.038817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwfw\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-kube-api-access-8rwfw\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.038817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-trusted-ca\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.038817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-tls\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.038817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-image-registry-private-configuration\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.039012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00a0b7b8-252a-4a58-86f6-0407fc0d3686-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zr89h\" (UID: \"00a0b7b8-252a-4a58-86f6-0407fc0d3686\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.039012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkp2\" (UniqueName: \"kubernetes.io/projected/47ceefb6-ebc1-4457-80b8-379f5182242f-kube-api-access-vqkp2\") pod \"managed-serviceaccount-addon-agent-77dff77b-mp9sk\" (UID: \"47ceefb6-ebc1-4457-80b8-379f5182242f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:16.039012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-bound-sa-token\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.039012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-ca-trust-extracted\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.039012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.038977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/00a0b7b8-252a-4a58-86f6-0407fc0d3686-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zr89h\" (UID: \"00a0b7b8-252a-4a58-86f6-0407fc0d3686\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.039012 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.039012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-installation-pull-secrets\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.039267 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.039041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47ceefb6-ebc1-4457-80b8-379f5182242f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77dff77b-mp9sk\" (UID: \"47ceefb6-ebc1-4457-80b8-379f5182242f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:16.040658 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.040030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-ca-trust-extracted\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.040658 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.040221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-certificates\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.040658 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.040619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-trusted-ca\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.042255 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.042225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47ceefb6-ebc1-4457-80b8-379f5182242f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77dff77b-mp9sk\" (UID: \"47ceefb6-ebc1-4457-80b8-379f5182242f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:16.043015 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.042910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-tls\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.043172 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.043149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-image-registry-private-configuration\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.043363 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.043340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-installation-pull-secrets\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.056716 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.056697 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8"] Sep 16 18:48:16.069515 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.069462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-667c9bc59c-n25ng"] Sep 16 18:48:16.072638 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.072616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwfw\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-kube-api-access-8rwfw\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.073545 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.073520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-bound-sa-token\") pod \"image-registry-7ff954774d-drwgj\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.074208 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.074191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkp2\" (UniqueName: \"kubernetes.io/projected/47ceefb6-ebc1-4457-80b8-379f5182242f-kube-api-access-vqkp2\") pod \"managed-serviceaccount-addon-agent-77dff77b-mp9sk\" (UID: \"47ceefb6-ebc1-4457-80b8-379f5182242f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:16.140367 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-image-registry-private-configuration\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140518 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-installation-pull-secrets\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140518 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00a0b7b8-252a-4a58-86f6-0407fc0d3686-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zr89h\" (UID: \"00a0b7b8-252a-4a58-86f6-0407fc0d3686\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.140518 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-trusted-ca\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140518 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07f271c7-4da0-46b9-8053-35e3f26a0ac3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g8b8\" (UID: \"07f271c7-4da0-46b9-8053-35e3f26a0ac3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:16.140518 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pvj\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-kube-api-access-l9pvj\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140708 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-registry-certificates\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140708 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/00a0b7b8-252a-4a58-86f6-0407fc0d3686-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zr89h\" (UID: \"00a0b7b8-252a-4a58-86f6-0407fc0d3686\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.140708 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-bound-sa-token\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140708 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-registry-tls\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.140857 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.140719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-ca-trust-extracted\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.141188 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.141172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/00a0b7b8-252a-4a58-86f6-0407fc0d3686-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zr89h\" (UID: \"00a0b7b8-252a-4a58-86f6-0407fc0d3686\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.142717 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.142697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00a0b7b8-252a-4a58-86f6-0407fc0d3686-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zr89h\" (UID: \"00a0b7b8-252a-4a58-86f6-0407fc0d3686\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.167088 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.167069 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" Sep 16 18:48:16.197360 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.197340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" Sep 16 18:48:16.241579 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-image-registry-private-configuration\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241579 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-installation-pull-secrets\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241751 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-trusted-ca\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241751 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07f271c7-4da0-46b9-8053-35e3f26a0ac3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g8b8\" (UID: \"07f271c7-4da0-46b9-8053-35e3f26a0ac3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:16.241751 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pvj\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-kube-api-access-l9pvj\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241905 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-registry-certificates\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241905 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-bound-sa-token\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241905 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-registry-tls\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.241905 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.241882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-ca-trust-extracted\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.242416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.242348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-ca-trust-extracted\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.243045 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.243009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-registry-certificates\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.244063 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.243668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-trusted-ca\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.244334 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.244315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-installation-pull-secrets\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.244818 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.244788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-registry-tls\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.245080 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.245058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-image-registry-private-configuration\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.245219 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.245188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07f271c7-4da0-46b9-8053-35e3f26a0ac3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g8b8\" (UID: \"07f271c7-4da0-46b9-8053-35e3f26a0ac3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:16.259695 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.259270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pvj\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-kube-api-access-l9pvj\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.259695 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.259678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2259add-edc3-4ba5-b6c4-29d4d68c30a6-bound-sa-token\") pod \"image-registry-667c9bc59c-n25ng\" (UID: \"b2259add-edc3-4ba5-b6c4-29d4d68c30a6\") " pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.292455 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.292437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:16.355629 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.355604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.444285 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.444262 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk"] Sep 16 18:48:16.447150 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:16.447130 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ceefb6_ebc1_4457_80b8_379f5182242f.slice/crio-c74537649e19706b726644cb1d250c1f03ef0653df2a9e2ca68b006de97f486a WatchSource:0}: Error finding container c74537649e19706b726644cb1d250c1f03ef0653df2a9e2ca68b006de97f486a: Status 404 returned error can't find the container with id c74537649e19706b726644cb1d250c1f03ef0653df2a9e2ca68b006de97f486a Sep 16 18:48:16.472713 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.472690 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h"] Sep 16 18:48:16.474917 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:16.474893 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a0b7b8_252a_4a58_86f6_0407fc0d3686.slice/crio-82ed93e1abcde6c9781bdd06bf76a83df98b3658b8824d91fad426ec33328f31 WatchSource:0}: Error finding container 82ed93e1abcde6c9781bdd06bf76a83df98b3658b8824d91fad426ec33328f31: Status 404 returned error can't find the container with id 82ed93e1abcde6c9781bdd06bf76a83df98b3658b8824d91fad426ec33328f31 Sep 16 18:48:16.559701 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.559685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8"] Sep 16 18:48:16.561508 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:16.561489 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f271c7_4da0_46b9_8053_35e3f26a0ac3.slice/crio-bff38520dad7828be131dd78c21c9dcafe06eab0815770e867e82c915cb480af WatchSource:0}: Error finding container bff38520dad7828be131dd78c21c9dcafe06eab0815770e867e82c915cb480af: Status 404 returned error can't find the container with id bff38520dad7828be131dd78c21c9dcafe06eab0815770e867e82c915cb480af Sep 16 18:48:16.626984 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.626938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-667c9bc59c-n25ng"] Sep 16 18:48:16.629567 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:16.629550 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2259add_edc3_4ba5_b6c4_29d4d68c30a6.slice/crio-02ffdb66dd51e48941708dcedec19ed9502d77b7b897795ffe63cc85e8235fce WatchSource:0}: Error finding container 02ffdb66dd51e48941708dcedec19ed9502d77b7b897795ffe63cc85e8235fce: Status 404 returned error can't find the container with id 02ffdb66dd51e48941708dcedec19ed9502d77b7b897795ffe63cc85e8235fce Sep 16 18:48:16.922896 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.922810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" event={"ID":"07f271c7-4da0-46b9-8053-35e3f26a0ac3","Type":"ContainerStarted","Data":"bff38520dad7828be131dd78c21c9dcafe06eab0815770e867e82c915cb480af"} Sep 16 18:48:16.924086 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.924053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" event={"ID":"47ceefb6-ebc1-4457-80b8-379f5182242f","Type":"ContainerStarted","Data":"c74537649e19706b726644cb1d250c1f03ef0653df2a9e2ca68b006de97f486a"} Sep 16 18:48:16.925242 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.925221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" event={"ID":"00a0b7b8-252a-4a58-86f6-0407fc0d3686","Type":"ContainerStarted","Data":"82ed93e1abcde6c9781bdd06bf76a83df98b3658b8824d91fad426ec33328f31"} Sep 16 18:48:16.927302 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.927285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wdmf" event={"ID":"9f2ac848-9cd2-42f2-97ae-74f25458627b","Type":"ContainerStarted","Data":"c4dbb319b83a31e3b4029cec246b7fe7d459bb09fcf0be85f310342dcabfbbc8"} Sep 16 18:48:16.928895 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.928875 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" event={"ID":"b2259add-edc3-4ba5-b6c4-29d4d68c30a6","Type":"ContainerStarted","Data":"d8097e255963efa3fb0e3467824e110386e8e67c1168855fdf278620afb76ea6"} Sep 16 18:48:16.928895 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.928891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.929040 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.928912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" event={"ID":"b2259add-edc3-4ba5-b6c4-29d4d68c30a6","Type":"ContainerStarted","Data":"02ffdb66dd51e48941708dcedec19ed9502d77b7b897795ffe63cc85e8235fce"} Sep 16 18:48:16.929040 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.928998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:16.935198 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.935178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:16.956199 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:16.956160 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6wdmf" podStartSLOduration=34.108506837 podStartE2EDuration="35.956147889s" totalImagesPullingTime="1.847641052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:16.953691365 +0000 UTC m=+160.076538869" watchObservedRunningTime="2026-09-16 18:48:16.956147889 +0000 UTC m=+160.078995393" Sep 16 18:48:17.010309 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.010261 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" podStartSLOduration=2.010240243 podStartE2EDuration="2.010240243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.009697968 +0000 UTC m=+160.132545473" watchObservedRunningTime="2026-09-16 18:48:17.010240243 +0000 UTC m=+160.133087749" Sep 16 18:48:17.049180 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049149 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-bound-sa-token\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049214 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-image-registry-private-configuration\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-ca-trust-extracted\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049276 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-installation-pull-secrets\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049301 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-trusted-ca\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049535 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049329 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-tls\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049535 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049359 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rwfw\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-kube-api-access-8rwfw\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049535 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049411 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-certificates\") pod \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\" (UID: \"a7f46fa1-1fab-46ef-b4be-a28cba8233e0\") " Sep 16 18:48:17.049694 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049541 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:48:17.049986 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049821 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-ca-trust-extracted\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.049986 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.049914 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:48:17.050855 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.050467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:48:17.052050 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.052012 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:48:17.052190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.052164 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:17.052237 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.052169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-kube-api-access-8rwfw" (OuterVolumeSpecName: "kube-api-access-8rwfw") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "kube-api-access-8rwfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:17.052739 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.052721 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:48:17.053003 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.052979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a7f46fa1-1fab-46ef-b4be-a28cba8233e0" (UID: "a7f46fa1-1fab-46ef-b4be-a28cba8233e0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:17.150580 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150548 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-image-registry-private-configuration\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.150580 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150578 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-installation-pull-secrets\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.150722 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150593 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-trusted-ca\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.150722 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150605 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.150722 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150615 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwfw\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-kube-api-access-8rwfw\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.150722 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150627 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-registry-certificates\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.150722 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.150639 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7f46fa1-1fab-46ef-b4be-a28cba8233e0-bound-sa-token\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:48:17.935830 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.935010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" event={"ID":"00a0b7b8-252a-4a58-86f6-0407fc0d3686","Type":"ContainerStarted","Data":"aa4d3d6cfdd02c3682fac8b158b684b6e57abd830ee0b39dabc8c873de3f65ce"} Sep 16 18:48:17.935830 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.935042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff954774d-drwgj" Sep 16 18:48:17.964686 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:17.964643 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zr89h" podStartSLOduration=1.657632865 podStartE2EDuration="2.964625114s" totalImagesPullingTime="1.306992249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.963662813 +0000 UTC m=+161.086510318" watchObservedRunningTime="2026-09-16 18:48:17.964625114 +0000 UTC m=+161.087472617" Sep 16 18:48:18.008568 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:18.008509 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ff954774d-drwgj"] Sep 16 18:48:18.011174 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:18.011148 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7ff954774d-drwgj"] Sep 16 18:48:19.270348 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.270314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:48:19.272708 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.272658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1985d5f3-1e2f-46aa-8688-e2491814291f-metrics-tls\") pod \"dns-default-bmgfk\" (UID: \"1985d5f3-1e2f-46aa-8688-e2491814291f\") " pod="openshift-dns/dns-default-bmgfk" Sep 16 18:48:19.400340 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.400313 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f46fa1-1fab-46ef-b4be-a28cba8233e0" path="/var/lib/kubelet/pods/a7f46fa1-1fab-46ef-b4be-a28cba8233e0/volumes" Sep 16 18:48:19.417227 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.417203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nqnnl\"" Sep 16 18:48:19.425434 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.425413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmgfk" Sep 16 18:48:19.704812 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.704788 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bmgfk"] Sep 16 18:48:19.706904 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:19.706881 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1985d5f3_1e2f_46aa_8688_e2491814291f.slice/crio-36c40026f24326639ed292e1deb36c6a54e9fd16baf863dc9c2a9a0b2c04936f WatchSource:0}: Error finding container 36c40026f24326639ed292e1deb36c6a54e9fd16baf863dc9c2a9a0b2c04936f: Status 404 returned error can't find the container with id 36c40026f24326639ed292e1deb36c6a54e9fd16baf863dc9c2a9a0b2c04936f Sep 16 18:48:19.943045 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.943013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmgfk" event={"ID":"1985d5f3-1e2f-46aa-8688-e2491814291f","Type":"ContainerStarted","Data":"36c40026f24326639ed292e1deb36c6a54e9fd16baf863dc9c2a9a0b2c04936f"} Sep 16 18:48:19.944436 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.944412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" event={"ID":"07f271c7-4da0-46b9-8053-35e3f26a0ac3","Type":"ContainerStarted","Data":"5001097f9ab1e4c50e11e998b076bf9305bfac1e0420fa8a8f401004a2f2b943"} Sep 16 18:48:19.944625 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.944594 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:19.945948 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.945928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" event={"ID":"47ceefb6-ebc1-4457-80b8-379f5182242f","Type":"ContainerStarted","Data":"94b74e1eeb991fa3b2110f0b3673d4a43b8053afe9b8e9ef05db3c4b5a1e1bd2"} Sep 16 18:48:19.949225 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.949206 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" Sep 16 18:48:19.970079 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:19.970011 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g8b8" podStartSLOduration=2.3543568280000002 podStartE2EDuration="4.970000246s" totalImagesPullingTime="2.615643418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:19.969972218 +0000 UTC m=+163.092819722" watchObservedRunningTime="2026-09-16 18:48:19.970000246 +0000 UTC m=+163.092847750" Sep 16 18:48:20.041346 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.041308 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dff77b-mp9sk" podStartSLOduration=2.309156229 podStartE2EDuration="5.041295403s" totalImagesPullingTime="2.732139174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:20.004046808 +0000 UTC m=+163.126894313" watchObservedRunningTime="2026-09-16 18:48:20.041295403 +0000 UTC m=+163.164142907" Sep 16 18:48:20.825969 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.825938 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5xmts"] Sep 16 18:48:20.829668 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.829635 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:20.832699 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.832673 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 18:48:20.832823 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.832675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 18:48:20.833051 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.833032 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 18:48:20.833138 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.833123 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hhrdr\"" Sep 16 18:48:20.833199 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.833159 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 18:48:20.833251 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.833159 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 18:48:20.846318 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.846300 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5xmts"] Sep 16 18:48:20.983172 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.983137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eba91c7-e9c2-4e25-b894-85ceaeee050b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:20.983344 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.983188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgw2\" (UniqueName: \"kubernetes.io/projected/5eba91c7-e9c2-4e25-b894-85ceaeee050b-kube-api-access-hfgw2\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:20.983344 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.983223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:20.983344 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:20.983296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.084489 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.084408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eba91c7-e9c2-4e25-b894-85ceaeee050b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.084489 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.084452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgw2\" (UniqueName: \"kubernetes.io/projected/5eba91c7-e9c2-4e25-b894-85ceaeee050b-kube-api-access-hfgw2\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.084664 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.084505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.084886 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.084862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.084991 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:48:21.084966 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 16 18:48:21.085060 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:48:21.085046 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-tls podName:5eba91c7-e9c2-4e25-b894-85ceaeee050b nodeName:}" failed. No retries permitted until 2026-09-16 18:48:21.585028618 +0000 UTC m=+164.707876106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-5xmts" (UID: "5eba91c7-e9c2-4e25-b894-85ceaeee050b") : secret "prometheus-operator-tls" not found Sep 16 18:48:21.085120 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.085087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eba91c7-e9c2-4e25-b894-85ceaeee050b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.087235 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.087211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.103396 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.103353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgw2\" (UniqueName: \"kubernetes.io/projected/5eba91c7-e9c2-4e25-b894-85ceaeee050b-kube-api-access-hfgw2\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.588808 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.588743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.590969 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.590952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eba91c7-e9c2-4e25-b894-85ceaeee050b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5xmts\" (UID: \"5eba91c7-e9c2-4e25-b894-85ceaeee050b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.740371 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.740347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" Sep 16 18:48:21.954026 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.953936 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmgfk" event={"ID":"1985d5f3-1e2f-46aa-8688-e2491814291f","Type":"ContainerStarted","Data":"4b18fc65dae64214f5a529499d2e8d38a1f1e57b242c7b363dae847d30509306"} Sep 16 18:48:21.954026 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.953995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmgfk" event={"ID":"1985d5f3-1e2f-46aa-8688-e2491814291f","Type":"ContainerStarted","Data":"fb1f44360ddf46eeb8ab456521bdd18b255440926a87c6daa4c87cf5832b5035"} Sep 16 18:48:21.954448 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.954100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bmgfk" Sep 16 18:48:21.976030 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:21.975982 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bmgfk" podStartSLOduration=129.47379681 podStartE2EDuration="2m10.975968995s" totalImagesPullingTime="1.502172185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:21.975470123 +0000 UTC m=+165.098317629" watchObservedRunningTime="2026-09-16 18:48:21.975968995 +0000 UTC m=+165.098816499" Sep 16 18:48:22.008543 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:22.008520 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5xmts"] Sep 16 18:48:22.011112 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:22.011088 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eba91c7_e9c2_4e25_b894_85ceaeee050b.slice/crio-5d8f947528c76d1286db182ed863599bd00620c6ac58cb3f2c64d0bc206f2943 WatchSource:0}: Error finding container 5d8f947528c76d1286db182ed863599bd00620c6ac58cb3f2c64d0bc206f2943: Status 404 returned error can't find the container with id 5d8f947528c76d1286db182ed863599bd00620c6ac58cb3f2c64d0bc206f2943 Sep 16 18:48:22.958701 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:22.958665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" event={"ID":"5eba91c7-e9c2-4e25-b894-85ceaeee050b","Type":"ContainerStarted","Data":"5d8f947528c76d1286db182ed863599bd00620c6ac58cb3f2c64d0bc206f2943"} Sep 16 18:48:23.963249 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:23.963209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" event={"ID":"5eba91c7-e9c2-4e25-b894-85ceaeee050b","Type":"ContainerStarted","Data":"d6d7609aff3254b3e976ae34fff6b05903ea3f296553792d0d76b2ce5065b27f"} Sep 16 18:48:23.963618 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:23.963260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" event={"ID":"5eba91c7-e9c2-4e25-b894-85ceaeee050b","Type":"ContainerStarted","Data":"2ea9e8d4d1001b51c247f38cd41d77e717d8556821efaabe195618d5939983ab"} Sep 16 18:48:23.987550 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:23.987507 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5xmts" podStartSLOduration=2.9545453 podStartE2EDuration="3.987491598s" totalImagesPullingTime="1.032946298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:23.986759208 +0000 UTC m=+167.109606713" watchObservedRunningTime="2026-09-16 18:48:23.987491598 +0000 UTC m=+167.110339134" Sep 16 18:48:26.396221 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.396193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:48:26.457552 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.457527 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-br5qf"] Sep 16 18:48:26.460834 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.460817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.463442 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.463421 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zt92r\"" Sep 16 18:48:26.463648 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.463630 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 18:48:26.463729 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.463633 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 18:48:26.464135 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.464120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 18:48:26.623497 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-sys\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623611 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-wtmp\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623611 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623611 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-textfile\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623745 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21654739-4f9a-4dff-8bac-be315398da1b-metrics-client-ca\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623745 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwndv\" (UniqueName: \"kubernetes.io/projected/21654739-4f9a-4dff-8bac-be315398da1b-kube-api-access-fwndv\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623745 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-tls\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623745 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.623887 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.623751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-root\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.724997 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.724947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-sys\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.724997 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.724978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-wtmp\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725106 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.724997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725106 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-sys\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725106 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-textfile\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725213 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21654739-4f9a-4dff-8bac-be315398da1b-metrics-client-ca\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725213 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwndv\" (UniqueName: \"kubernetes.io/projected/21654739-4f9a-4dff-8bac-be315398da1b-kube-api-access-fwndv\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725213 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-wtmp\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725213 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-tls\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725213 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725472 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-root\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725472 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21654739-4f9a-4dff-8bac-be315398da1b-root\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725563 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-textfile\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725755 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21654739-4f9a-4dff-8bac-be315398da1b-metrics-client-ca\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.725795 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.725740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.727854 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.727829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.728021 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.728003 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21654739-4f9a-4dff-8bac-be315398da1b-node-exporter-tls\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.738694 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.738676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwndv\" (UniqueName: \"kubernetes.io/projected/21654739-4f9a-4dff-8bac-be315398da1b-kube-api-access-fwndv\") pod \"node-exporter-br5qf\" (UID: \"21654739-4f9a-4dff-8bac-be315398da1b\") " pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.769322 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.769301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-br5qf" Sep 16 18:48:26.780489 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:26.780468 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21654739_4f9a_4dff_8bac_be315398da1b.slice/crio-2c31437eec1340c07c567db0167c0c9af957e51eed19c7a5d6916dd4c20ef072 WatchSource:0}: Error finding container 2c31437eec1340c07c567db0167c0c9af957e51eed19c7a5d6916dd4c20ef072: Status 404 returned error can't find the container with id 2c31437eec1340c07c567db0167c0c9af957e51eed19c7a5d6916dd4c20ef072 Sep 16 18:48:26.973834 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:26.973807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br5qf" event={"ID":"21654739-4f9a-4dff-8bac-be315398da1b","Type":"ContainerStarted","Data":"2c31437eec1340c07c567db0167c0c9af957e51eed19c7a5d6916dd4c20ef072"} Sep 16 18:48:27.979244 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:27.979186 2580 generic.go:356] "Generic (PLEG): container finished" podID="21654739-4f9a-4dff-8bac-be315398da1b" containerID="5b9ab41229df3a72913353757771bdd5c5d467ef91855a2ffb9c2860e0ac479e" exitCode=0 Sep 16 18:48:27.979244 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:27.979236 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br5qf" event={"ID":"21654739-4f9a-4dff-8bac-be315398da1b","Type":"ContainerDied","Data":"5b9ab41229df3a72913353757771bdd5c5d467ef91855a2ffb9c2860e0ac479e"} Sep 16 18:48:28.984816 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:28.984747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br5qf" event={"ID":"21654739-4f9a-4dff-8bac-be315398da1b","Type":"ContainerStarted","Data":"88103f2ef7567864c03208c838bc2c106075b538d666a66c9d55a6c2686cba78"} Sep 16 18:48:28.984816 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:28.984794 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br5qf" event={"ID":"21654739-4f9a-4dff-8bac-be315398da1b","Type":"ContainerStarted","Data":"ef1c5162b8e66de4ffc19fc51e52148bee2dda8e1fb89ce381d9331ab647bd17"} Sep 16 18:48:29.013210 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.013165 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-br5qf" podStartSLOduration=2.078806532 podStartE2EDuration="3.013149829s" totalImagesPullingTime="867.55752ms" totalInitContainerRuntime="66.785777ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:29.012209092 +0000 UTC m=+172.135056596" watchObservedRunningTime="2026-09-16 18:48:29.013149829 +0000 UTC m=+172.135997316" Sep 16 18:48:29.378271 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.378234 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d"] Sep 16 18:48:29.381794 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.381772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.385783 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.385760 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n9527\"" Sep 16 18:48:29.386494 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.386067 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 18:48:29.386494 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.386172 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 18:48:29.386494 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.386199 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 18:48:29.386494 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.386302 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 18:48:29.386494 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.386374 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7gfbe3p9rmhro\"" Sep 16 18:48:29.386760 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.386511 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 18:48:29.409416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.409363 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d"] Sep 16 18:48:29.546535 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-metrics-client-ca\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546660 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddqv\" (UniqueName: \"kubernetes.io/projected/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-kube-api-access-2ddqv\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546660 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546743 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546743 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-grpc-tls\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546812 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-tls\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546812 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.546812 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.546801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.647836 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-tls\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.647836 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.647836 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.648035 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-metrics-client-ca\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.648035 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddqv\" (UniqueName: \"kubernetes.io/projected/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-kube-api-access-2ddqv\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.648035 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.648035 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.648035 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.647954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-grpc-tls\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.648849 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.648825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-metrics-client-ca\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.650466 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.650448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-tls\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.650466 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.650457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.650881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.650855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.650969 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.650954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.651011 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.650981 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.651289 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.651270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-secret-grpc-tls\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.656263 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.656242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddqv\" (UniqueName: \"kubernetes.io/projected/f7f05bca-1cde-46b6-a40a-08b27aa00f4d-kube-api-access-2ddqv\") pod \"thanos-querier-77ffbbd67d-vrc4d\" (UID: \"f7f05bca-1cde-46b6-a40a-08b27aa00f4d\") " pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.695042 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.695021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:29.972549 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.972527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d"] Sep 16 18:48:29.974272 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:29.974247 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f05bca_1cde_46b6_a40a_08b27aa00f4d.slice/crio-f6978966566d750ba413ea878998924ffa26bd02ef8f055e981de0170a057f59 WatchSource:0}: Error finding container f6978966566d750ba413ea878998924ffa26bd02ef8f055e981de0170a057f59: Status 404 returned error can't find the container with id f6978966566d750ba413ea878998924ffa26bd02ef8f055e981de0170a057f59 Sep 16 18:48:29.990156 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:29.990127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"f6978966566d750ba413ea878998924ffa26bd02ef8f055e981de0170a057f59"} Sep 16 18:48:31.961258 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:31.961234 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bmgfk" Sep 16 18:48:32.000167 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:32.000138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"44caa2049305f46363c5298e5e9b33948b5b9e77e76ad48e50ea543f860c791d"} Sep 16 18:48:32.000253 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:32.000200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"a80d38e9a84b954226a3ac50438cea2931edc7f511e15ddb0416a32414da88c7"} Sep 16 18:48:32.000253 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:32.000226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"46c3535aaa7de82835f52b0d57b9c4d7224b169aff7db950b1f45048dfb54217"} Sep 16 18:48:33.007030 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:33.006941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"2139bbc8ef797d44c9d8d0b3eeb0d1003b388ae2d33d4ad821e2d6b750aa6834"} Sep 16 18:48:33.007488 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:33.007054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"a8a8fc9c2b4065fb4076e4f4274c3d035a37b3f9026f7e2616580ee0ec866593"} Sep 16 18:48:33.007488 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:33.007091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" event={"ID":"f7f05bca-1cde-46b6-a40a-08b27aa00f4d","Type":"ContainerStarted","Data":"5641e9603cdbade5d8bf70b32778c779f791d12f4380b0264adab24027b8bea9"} Sep 16 18:48:33.007488 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:33.007302 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:33.032000 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:33.031945 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" podStartSLOduration=1.58248425 podStartE2EDuration="4.031921964s" totalImagesPullingTime="2.449437714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:33.030919439 +0000 UTC m=+176.153766944" watchObservedRunningTime="2026-09-16 18:48:33.031921964 +0000 UTC m=+176.154769468" Sep 16 18:48:36.359934 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:36.359903 2580 patch_prober.go:28] interesting pod/image-registry-667c9bc59c-n25ng container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 16 18:48:36.360303 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:36.359957 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" podUID="b2259add-edc3-4ba5-b6c4-29d4d68c30a6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:48:37.939699 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:37.939670 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-667c9bc59c-n25ng" Sep 16 18:48:38.949510 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.949482 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbc8fb76f-r4nkx"] Sep 16 18:48:38.955751 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.955730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:38.958253 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958230 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 18:48:38.958355 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958309 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 18:48:38.958355 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958308 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 18:48:38.958945 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6gpst\"" Sep 16 18:48:38.959091 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958927 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 18:48:38.959194 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958938 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 18:48:38.959194 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.958994 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 18:48:38.959194 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.959036 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 18:48:38.963603 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.963583 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 18:48:38.964671 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:38.964652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbc8fb76f-r4nkx"] Sep 16 18:48:39.015720 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.015702 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77ffbbd67d-vrc4d" Sep 16 18:48:39.121152 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-oauth-config\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.121152 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw22x\" (UniqueName: \"kubernetes.io/projected/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-kube-api-access-gw22x\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.121314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-config\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.121314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-service-ca\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.121314 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-serving-cert\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.121474 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-trusted-ca-bundle\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.121508 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.121496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-oauth-serving-cert\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.222943 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.222880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-oauth-config\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.222943 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.222905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw22x\" (UniqueName: \"kubernetes.io/projected/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-kube-api-access-gw22x\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.222943 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.222928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-config\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.223126 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.222951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-service-ca\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.223126 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.222981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-serving-cert\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.223126 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.223017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-trusted-ca-bundle\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.223126 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.223052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-oauth-serving-cert\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.223758 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.223735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-config\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.223991 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.223970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-trusted-ca-bundle\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.224071 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.224007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-service-ca\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.224124 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.224101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-oauth-serving-cert\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.225394 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.225362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-serving-cert\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.225465 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.225416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-oauth-config\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.231239 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.231219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw22x\" (UniqueName: \"kubernetes.io/projected/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-kube-api-access-gw22x\") pod \"console-cbc8fb76f-r4nkx\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.266158 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.266136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:39.551251 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:39.551223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbc8fb76f-r4nkx"] Sep 16 18:48:39.553345 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:39.553316 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb2dd0b_04c0_4417_a3df_02a97a14a0a7.slice/crio-45ccfeb69d5b03784593e3b6cd1ad1d0a8057964bebbba09d572260d2e4be952 WatchSource:0}: Error finding container 45ccfeb69d5b03784593e3b6cd1ad1d0a8057964bebbba09d572260d2e4be952: Status 404 returned error can't find the container with id 45ccfeb69d5b03784593e3b6cd1ad1d0a8057964bebbba09d572260d2e4be952 Sep 16 18:48:40.033451 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:40.033361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc8fb76f-r4nkx" event={"ID":"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7","Type":"ContainerStarted","Data":"45ccfeb69d5b03784593e3b6cd1ad1d0a8057964bebbba09d572260d2e4be952"} Sep 16 18:48:43.047234 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:43.047194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc8fb76f-r4nkx" event={"ID":"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7","Type":"ContainerStarted","Data":"343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a"} Sep 16 18:48:43.069964 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:43.069922 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cbc8fb76f-r4nkx" podStartSLOduration=2.310601266 podStartE2EDuration="5.069908423s" totalImagesPullingTime="2.759307157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.068473549 +0000 UTC m=+186.191321051" watchObservedRunningTime="2026-09-16 18:48:43.069908423 +0000 UTC m=+186.192755927" Sep 16 18:48:43.953525 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:43.953498 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbc8fb76f-r4nkx"] Sep 16 18:48:43.998528 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:43.998502 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-567c4b645d-6wpkn"] Sep 16 18:48:44.001846 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.001829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.013114 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.013093 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567c4b645d-6wpkn"] Sep 16 18:48:44.062185 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-trusted-ca-bundle\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.062522 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-config\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.062522 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-service-ca\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.062522 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-oauth-config\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.062522 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-oauth-serving-cert\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.062522 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-serving-cert\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.062743 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.062553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf78\" (UniqueName: \"kubernetes.io/projected/76116bc0-e617-4a4b-9e45-4af9c0b1e423-kube-api-access-7vf78\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163571 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf78\" (UniqueName: \"kubernetes.io/projected/76116bc0-e617-4a4b-9e45-4af9c0b1e423-kube-api-access-7vf78\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163681 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-trusted-ca-bundle\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163744 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-config\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163744 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-service-ca\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163849 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-oauth-config\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163849 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-oauth-serving-cert\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.163948 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.163849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-serving-cert\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.164405 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.164367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-service-ca\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.164512 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.164485 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-config\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.164560 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.164490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-trusted-ca-bundle\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.164560 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.164528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-oauth-serving-cert\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.167252 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.167231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-serving-cert\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.167252 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.167245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-oauth-config\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.172628 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.172607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf78\" (UniqueName: \"kubernetes.io/projected/76116bc0-e617-4a4b-9e45-4af9c0b1e423-kube-api-access-7vf78\") pod \"console-567c4b645d-6wpkn\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.310295 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.310273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:44.583302 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:44.583159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567c4b645d-6wpkn"] Sep 16 18:48:44.585993 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:48:44.585968 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76116bc0_e617_4a4b_9e45_4af9c0b1e423.slice/crio-9d669917bcd5b7cc09387e618db3db0a74518b14ef24f3dd36a123563b004fb2 WatchSource:0}: Error finding container 9d669917bcd5b7cc09387e618db3db0a74518b14ef24f3dd36a123563b004fb2: Status 404 returned error can't find the container with id 9d669917bcd5b7cc09387e618db3db0a74518b14ef24f3dd36a123563b004fb2 Sep 16 18:48:45.056656 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:45.056615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567c4b645d-6wpkn" event={"ID":"76116bc0-e617-4a4b-9e45-4af9c0b1e423","Type":"ContainerStarted","Data":"3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19"} Sep 16 18:48:45.056811 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:45.056710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567c4b645d-6wpkn" event={"ID":"76116bc0-e617-4a4b-9e45-4af9c0b1e423","Type":"ContainerStarted","Data":"9d669917bcd5b7cc09387e618db3db0a74518b14ef24f3dd36a123563b004fb2"} Sep 16 18:48:45.119631 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:45.119585 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-567c4b645d-6wpkn" podStartSLOduration=2.11956841 podStartE2EDuration="2.11956841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:45.117333414 +0000 UTC m=+188.240180919" watchObservedRunningTime="2026-09-16 18:48:45.11956841 +0000 UTC m=+188.242415914" Sep 16 18:48:49.266589 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:49.266558 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:48:54.311116 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:54.311083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:54.311598 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:54.311144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:54.315954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:54.315922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:48:55.095630 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:48:55.095606 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:49:01.113577 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:01.113552 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9a248c6-76bc-457d-a2b8-c4b5231a04bb" containerID="f01aef75dd658409d9afd6b2ffc01198edc431ffc5af8082cecc25129704014a" exitCode=0 Sep 16 18:49:01.113868 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:01.113632 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qdlfp" event={"ID":"f9a248c6-76bc-457d-a2b8-c4b5231a04bb","Type":"ContainerDied","Data":"f01aef75dd658409d9afd6b2ffc01198edc431ffc5af8082cecc25129704014a"} Sep 16 18:49:01.113971 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:01.113958 2580 scope.go:117] "RemoveContainer" containerID="f01aef75dd658409d9afd6b2ffc01198edc431ffc5af8082cecc25129704014a" Sep 16 18:49:02.119123 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:02.119090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qdlfp" event={"ID":"f9a248c6-76bc-457d-a2b8-c4b5231a04bb","Type":"ContainerStarted","Data":"0e50ecfc3827676874f6e5d4147c46eac9cb0676b591efa1709d487143226ba0"} Sep 16 18:49:10.076117 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.076065 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cbc8fb76f-r4nkx" podUID="aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" containerName="console" containerID="cri-o://343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a" gracePeriod=15 Sep 16 18:49:10.315102 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.315080 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:49:10.356923 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.356866 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-serving-cert\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.356923 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.356917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-service-ca\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.357070 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.356944 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-config\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.357070 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.356969 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-trusted-ca-bundle\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.357070 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw22x\" (UniqueName: \"kubernetes.io/projected/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-kube-api-access-gw22x\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.357210 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-oauth-config\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.357210 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357108 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-oauth-serving-cert\") pod \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\" (UID: \"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7\") " Sep 16 18:49:10.357310 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357281 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-service-ca" (OuterVolumeSpecName: "service-ca") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.357371 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357341 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-service-ca\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.357371 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357345 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-config" (OuterVolumeSpecName: "console-config") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.357570 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357536 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.357675 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.357634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.359077 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.359055 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:10.359395 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.359360 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-kube-api-access-gw22x" (OuterVolumeSpecName: "kube-api-access-gw22x") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "kube-api-access-gw22x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:10.359665 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.359648 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" (UID: "aeb2dd0b-04c0-4417-a3df-02a97a14a0a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:10.458531 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.458504 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-serving-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.458531 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.458528 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.458659 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.458536 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-trusted-ca-bundle\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.458659 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.458545 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw22x\" (UniqueName: \"kubernetes.io/projected/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-kube-api-access-gw22x\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.458659 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.458554 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-console-oauth-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.458659 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:10.458561 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7-oauth-serving-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:49:11.152708 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.152679 2580 generic.go:356] "Generic (PLEG): container finished" podID="aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" containerID="343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a" exitCode=0 Sep 16 18:49:11.153063 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.152720 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc8fb76f-r4nkx" Sep 16 18:49:11.153063 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.152747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc8fb76f-r4nkx" event={"ID":"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7","Type":"ContainerDied","Data":"343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a"} Sep 16 18:49:11.153063 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.152784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc8fb76f-r4nkx" event={"ID":"aeb2dd0b-04c0-4417-a3df-02a97a14a0a7","Type":"ContainerDied","Data":"45ccfeb69d5b03784593e3b6cd1ad1d0a8057964bebbba09d572260d2e4be952"} Sep 16 18:49:11.153063 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.152803 2580 scope.go:117] "RemoveContainer" containerID="343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a" Sep 16 18:49:11.160932 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.160915 2580 scope.go:117] "RemoveContainer" containerID="343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a" Sep 16 18:49:11.161193 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:49:11.161176 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a\": container with ID starting with 343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a not found: ID does not exist" containerID="343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a" Sep 16 18:49:11.161237 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.161201 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a"} err="failed to get container status \"343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a\": rpc error: code = NotFound desc = could not find container \"343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a\": container with ID starting with 343aa0fedec230f06cb4bd35d249c56cbd7173e609451f30f19a3164a9a4934a not found: ID does not exist" Sep 16 18:49:11.180031 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.179991 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbc8fb76f-r4nkx"] Sep 16 18:49:11.191100 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.187249 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbc8fb76f-r4nkx"] Sep 16 18:49:11.401070 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:11.401039 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" path="/var/lib/kubelet/pods/aeb2dd0b-04c0-4417-a3df-02a97a14a0a7/volumes" Sep 16 18:49:49.176877 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:49.176836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:49:49.179400 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:49.179350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593cdcf1-d960-4800-a58f-46e20476cb32-metrics-certs\") pod \"network-metrics-daemon-8tzg8\" (UID: \"593cdcf1-d960-4800-a58f-46e20476cb32\") " pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:49:49.199977 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:49.199950 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pl7qs\"" Sep 16 18:49:49.207960 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:49.207938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tzg8" Sep 16 18:49:49.477406 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:49.477361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8tzg8"] Sep 16 18:49:49.480049 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:49:49.480020 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593cdcf1_d960_4800_a58f_46e20476cb32.slice/crio-63319959f6201d3dfe0efb82c022cfba4ffba272f8172b60fd7b64f096699bf6 WatchSource:0}: Error finding container 63319959f6201d3dfe0efb82c022cfba4ffba272f8172b60fd7b64f096699bf6: Status 404 returned error can't find the container with id 63319959f6201d3dfe0efb82c022cfba4ffba272f8172b60fd7b64f096699bf6 Sep 16 18:49:50.309451 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:50.309414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8tzg8" event={"ID":"593cdcf1-d960-4800-a58f-46e20476cb32","Type":"ContainerStarted","Data":"63319959f6201d3dfe0efb82c022cfba4ffba272f8172b60fd7b64f096699bf6"} Sep 16 18:49:51.314894 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:51.314856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8tzg8" event={"ID":"593cdcf1-d960-4800-a58f-46e20476cb32","Type":"ContainerStarted","Data":"6679c7fa712ee564d2ab059f04930aebf003f99839f94cacf4cb24342da073dd"} Sep 16 18:49:51.315328 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:51.314909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8tzg8" event={"ID":"593cdcf1-d960-4800-a58f-46e20476cb32","Type":"ContainerStarted","Data":"f5d02ea9e4411837cb83c3280db0fb7cd00e6bdb85c3391a15022f1e32729b21"} Sep 16 18:49:51.339399 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:51.339339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8tzg8" podStartSLOduration=253.499477632 podStartE2EDuration="4m14.339323882s" totalImagesPullingTime="839.84625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:51.33784031 +0000 UTC m=+254.460687817" watchObservedRunningTime="2026-09-16 18:49:51.339323882 +0000 UTC m=+254.462171388" Sep 16 18:49:52.747433 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.747406 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bd87c846-78cvj"] Sep 16 18:49:52.747860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.747728 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" containerName="console" Sep 16 18:49:52.747860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.747741 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" containerName="console" Sep 16 18:49:52.747860 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.747805 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb2dd0b-04c0-4417-a3df-02a97a14a0a7" containerName="console" Sep 16 18:49:52.751016 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.750995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.765416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.765361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bd87c846-78cvj"] Sep 16 18:49:52.807015 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.806966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-oauth-serving-cert\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.807196 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.807033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-service-ca\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.807196 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.807059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsr4k\" (UniqueName: \"kubernetes.io/projected/4f20be76-079d-4772-8c17-50a2012848bb-kube-api-access-gsr4k\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.807196 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.807114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-serving-cert\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.807196 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.807172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-trusted-ca-bundle\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.807434 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.807200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-oauth-config\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.807434 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.807256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-console-config\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.907941 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.907904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-console-config\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.907941 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.907942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-oauth-serving-cert\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.907965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-service-ca\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.907988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsr4k\" (UniqueName: \"kubernetes.io/projected/4f20be76-079d-4772-8c17-50a2012848bb-kube-api-access-gsr4k\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-serving-cert\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-trusted-ca-bundle\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908190 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-oauth-config\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908791 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-console-config\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.908965 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-oauth-serving-cert\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.909058 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-service-ca\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.909058 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.908965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-trusted-ca-bundle\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.910610 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.910588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-oauth-config\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.910683 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.910644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-serving-cert\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:52.916668 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:52.916644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsr4k\" (UniqueName: \"kubernetes.io/projected/4f20be76-079d-4772-8c17-50a2012848bb-kube-api-access-gsr4k\") pod \"console-55bd87c846-78cvj\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:53.061343 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:53.061308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:49:53.342448 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:53.342352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bd87c846-78cvj"] Sep 16 18:49:53.345410 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:49:53.345365 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f20be76_079d_4772_8c17_50a2012848bb.slice/crio-400cfc512d13933a2429f78e4b9f435691c58a795d32e822b40c99088685a862 WatchSource:0}: Error finding container 400cfc512d13933a2429f78e4b9f435691c58a795d32e822b40c99088685a862: Status 404 returned error can't find the container with id 400cfc512d13933a2429f78e4b9f435691c58a795d32e822b40c99088685a862 Sep 16 18:49:54.327797 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:54.327757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd87c846-78cvj" event={"ID":"4f20be76-079d-4772-8c17-50a2012848bb","Type":"ContainerStarted","Data":"5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a"} Sep 16 18:49:54.328347 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:54.327812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd87c846-78cvj" event={"ID":"4f20be76-079d-4772-8c17-50a2012848bb","Type":"ContainerStarted","Data":"400cfc512d13933a2429f78e4b9f435691c58a795d32e822b40c99088685a862"} Sep 16 18:49:54.347536 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:49:54.347481 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55bd87c846-78cvj" podStartSLOduration=2.347463682 podStartE2EDuration="2.347463682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:54.346082355 +0000 UTC m=+257.468929862" watchObservedRunningTime="2026-09-16 18:49:54.347463682 +0000 UTC m=+257.470311187" Sep 16 18:50:03.061620 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:03.061529 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:50:03.061620 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:03.061590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:50:03.066293 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:03.066270 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:50:03.366404 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:03.366314 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:50:03.418729 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:03.418694 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567c4b645d-6wpkn"] Sep 16 18:50:28.441758 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.441699 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-567c4b645d-6wpkn" podUID="76116bc0-e617-4a4b-9e45-4af9c0b1e423" containerName="console" containerID="cri-o://3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19" gracePeriod=15 Sep 16 18:50:28.679641 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.679620 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:50:28.795110 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795082 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-oauth-serving-cert\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-config\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795153 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-serving-cert\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-trusted-ca-bundle\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795223 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-service-ca\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-oauth-config\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795532 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795285 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vf78\" (UniqueName: \"kubernetes.io/projected/76116bc0-e617-4a4b-9e45-4af9c0b1e423-kube-api-access-7vf78\") pod \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\" (UID: \"76116bc0-e617-4a4b-9e45-4af9c0b1e423\") " Sep 16 18:50:28.795592 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:28.795695 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795575 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-config" (OuterVolumeSpecName: "console-config") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:28.795817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795719 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-service-ca" (OuterVolumeSpecName: "service-ca") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:28.795817 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.795740 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:28.797723 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.797687 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:50:28.797807 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.797759 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:50:28.797807 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.797769 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76116bc0-e617-4a4b-9e45-4af9c0b1e423-kube-api-access-7vf78" (OuterVolumeSpecName: "kube-api-access-7vf78") pod "76116bc0-e617-4a4b-9e45-4af9c0b1e423" (UID: "76116bc0-e617-4a4b-9e45-4af9c0b1e423"). InnerVolumeSpecName "kube-api-access-7vf78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:50:28.896743 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896716 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-oauth-serving-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:28.896743 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896745 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:28.896951 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896758 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-serving-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:28.896951 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896789 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-trusted-ca-bundle\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:28.896951 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896804 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76116bc0-e617-4a4b-9e45-4af9c0b1e423-service-ca\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:28.896951 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896816 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76116bc0-e617-4a4b-9e45-4af9c0b1e423-console-oauth-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:28.896951 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:28.896828 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vf78\" (UniqueName: \"kubernetes.io/projected/76116bc0-e617-4a4b-9e45-4af9c0b1e423-kube-api-access-7vf78\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:50:29.458391 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.458356 2580 generic.go:356] "Generic (PLEG): container finished" podID="76116bc0-e617-4a4b-9e45-4af9c0b1e423" containerID="3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19" exitCode=0 Sep 16 18:50:29.458801 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.458415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567c4b645d-6wpkn" event={"ID":"76116bc0-e617-4a4b-9e45-4af9c0b1e423","Type":"ContainerDied","Data":"3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19"} Sep 16 18:50:29.458801 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.458429 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567c4b645d-6wpkn" Sep 16 18:50:29.458801 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.458465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567c4b645d-6wpkn" event={"ID":"76116bc0-e617-4a4b-9e45-4af9c0b1e423","Type":"ContainerDied","Data":"9d669917bcd5b7cc09387e618db3db0a74518b14ef24f3dd36a123563b004fb2"} Sep 16 18:50:29.458801 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.458494 2580 scope.go:117] "RemoveContainer" containerID="3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19" Sep 16 18:50:29.466750 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.466732 2580 scope.go:117] "RemoveContainer" containerID="3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19" Sep 16 18:50:29.467019 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:50:29.467000 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19\": container with ID starting with 3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19 not found: ID does not exist" containerID="3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19" Sep 16 18:50:29.467086 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.467026 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19"} err="failed to get container status \"3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19\": rpc error: code = NotFound desc = could not find container \"3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19\": container with ID starting with 3cfcfc31130904fafa469cb0ef69ddb81cddb4af3b4485af293e318cf6577e19 not found: ID does not exist" Sep 16 18:50:29.475864 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.475842 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567c4b645d-6wpkn"] Sep 16 18:50:29.480556 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:29.480537 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-567c4b645d-6wpkn"] Sep 16 18:50:31.401455 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:31.401428 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76116bc0-e617-4a4b-9e45-4af9c0b1e423" path="/var/lib/kubelet/pods/76116bc0-e617-4a4b-9e45-4af9c0b1e423/volumes" Sep 16 18:50:37.267873 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:37.267847 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:50:37.274257 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:37.274235 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:50:37.279797 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:50:37.279773 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 18:52:26.128078 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.128040 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv"] Sep 16 18:52:26.128674 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.128480 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76116bc0-e617-4a4b-9e45-4af9c0b1e423" containerName="console" Sep 16 18:52:26.128674 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.128496 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76116bc0-e617-4a4b-9e45-4af9c0b1e423" containerName="console" Sep 16 18:52:26.128674 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.128568 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76116bc0-e617-4a4b-9e45-4af9c0b1e423" containerName="console" Sep 16 18:52:26.131310 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.131289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.134995 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.134977 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-ms9rr\"" Sep 16 18:52:26.135095 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.135051 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 18:52:26.135347 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.135322 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 18:52:26.135469 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.135405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 18:52:26.150851 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.150825 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv"] Sep 16 18:52:26.187263 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.187239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg7f\" (UniqueName: \"kubernetes.io/projected/5d0d99c7-0beb-4388-b26a-da27d69f3747-kube-api-access-2tg7f\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mzqkv\" (UID: \"5d0d99c7-0beb-4388-b26a-da27d69f3747\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.187359 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.187279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5d0d99c7-0beb-4388-b26a-da27d69f3747-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mzqkv\" (UID: \"5d0d99c7-0beb-4388-b26a-da27d69f3747\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.288563 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.288536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg7f\" (UniqueName: \"kubernetes.io/projected/5d0d99c7-0beb-4388-b26a-da27d69f3747-kube-api-access-2tg7f\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mzqkv\" (UID: \"5d0d99c7-0beb-4388-b26a-da27d69f3747\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.288665 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.288583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5d0d99c7-0beb-4388-b26a-da27d69f3747-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mzqkv\" (UID: \"5d0d99c7-0beb-4388-b26a-da27d69f3747\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.290954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.290934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5d0d99c7-0beb-4388-b26a-da27d69f3747-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mzqkv\" (UID: \"5d0d99c7-0beb-4388-b26a-da27d69f3747\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.298513 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.298495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg7f\" (UniqueName: \"kubernetes.io/projected/5d0d99c7-0beb-4388-b26a-da27d69f3747-kube-api-access-2tg7f\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mzqkv\" (UID: \"5d0d99c7-0beb-4388-b26a-da27d69f3747\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.441093 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.441047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:26.717728 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.717561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv"] Sep 16 18:52:26.720606 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:52:26.720578 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0d99c7_0beb_4388_b26a_da27d69f3747.slice/crio-9b99ba9035fda3b19674b41349f1c207c4702e24df3025d7050cb8273b6dedc2 WatchSource:0}: Error finding container 9b99ba9035fda3b19674b41349f1c207c4702e24df3025d7050cb8273b6dedc2: Status 404 returned error can't find the container with id 9b99ba9035fda3b19674b41349f1c207c4702e24df3025d7050cb8273b6dedc2 Sep 16 18:52:26.722428 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.722406 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:52:26.877163 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:26.877128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" event={"ID":"5d0d99c7-0beb-4388-b26a-da27d69f3747","Type":"ContainerStarted","Data":"9b99ba9035fda3b19674b41349f1c207c4702e24df3025d7050cb8273b6dedc2"} Sep 16 18:52:30.900153 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:30.900118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" event={"ID":"5d0d99c7-0beb-4388-b26a-da27d69f3747","Type":"ContainerStarted","Data":"d9b96ba534c3360f5f63d011d270d9a6c6362000ad83e4076cb51513a9c32ae9"} Sep 16 18:52:30.900577 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:30.900230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:30.960039 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:30.959992 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" podStartSLOduration=1.80656778 podStartE2EDuration="4.959971412s" totalImagesPullingTime="3.153403632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:30.958809467 +0000 UTC m=+414.081656974" watchObservedRunningTime="2026-09-16 18:52:30.959971412 +0000 UTC m=+414.082818917" Sep 16 18:52:31.819112 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.819080 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6s47q"] Sep 16 18:52:31.827023 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.826901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:31.830757 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.830647 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 18:52:31.830913 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.830827 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 16 18:52:31.831570 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.831548 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tgqkq\"" Sep 16 18:52:31.840777 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.840745 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6s47q"] Sep 16 18:52:31.931347 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.931319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3175fc93-fb32-455d-a8be-79d33e2eafab-certificates\") pod \"keda-admission-6c4d864b86-6s47q\" (UID: \"3175fc93-fb32-455d-a8be-79d33e2eafab\") " pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:31.931739 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:31.931406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntqd\" (UniqueName: \"kubernetes.io/projected/3175fc93-fb32-455d-a8be-79d33e2eafab-kube-api-access-tntqd\") pod \"keda-admission-6c4d864b86-6s47q\" (UID: \"3175fc93-fb32-455d-a8be-79d33e2eafab\") " pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:32.032167 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.032132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntqd\" (UniqueName: \"kubernetes.io/projected/3175fc93-fb32-455d-a8be-79d33e2eafab-kube-api-access-tntqd\") pod \"keda-admission-6c4d864b86-6s47q\" (UID: \"3175fc93-fb32-455d-a8be-79d33e2eafab\") " pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:32.032167 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.032173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3175fc93-fb32-455d-a8be-79d33e2eafab-certificates\") pod \"keda-admission-6c4d864b86-6s47q\" (UID: \"3175fc93-fb32-455d-a8be-79d33e2eafab\") " pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:32.035416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.035362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3175fc93-fb32-455d-a8be-79d33e2eafab-certificates\") pod \"keda-admission-6c4d864b86-6s47q\" (UID: \"3175fc93-fb32-455d-a8be-79d33e2eafab\") " pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:32.042231 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.042199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntqd\" (UniqueName: \"kubernetes.io/projected/3175fc93-fb32-455d-a8be-79d33e2eafab-kube-api-access-tntqd\") pod \"keda-admission-6c4d864b86-6s47q\" (UID: \"3175fc93-fb32-455d-a8be-79d33e2eafab\") " pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:32.140086 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.140015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:32.419349 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.419269 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6s47q"] Sep 16 18:52:32.422494 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:52:32.422465 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175fc93_fb32_455d_a8be_79d33e2eafab.slice/crio-b097c568dddb4710ccb3a8b4bd6c827477c7724c0d03ccb0f24462b7695431ff WatchSource:0}: Error finding container b097c568dddb4710ccb3a8b4bd6c827477c7724c0d03ccb0f24462b7695431ff: Status 404 returned error can't find the container with id b097c568dddb4710ccb3a8b4bd6c827477c7724c0d03ccb0f24462b7695431ff Sep 16 18:52:32.909414 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:32.909362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" event={"ID":"3175fc93-fb32-455d-a8be-79d33e2eafab","Type":"ContainerStarted","Data":"b097c568dddb4710ccb3a8b4bd6c827477c7724c0d03ccb0f24462b7695431ff"} Sep 16 18:52:34.920971 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:34.920932 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" event={"ID":"3175fc93-fb32-455d-a8be-79d33e2eafab","Type":"ContainerStarted","Data":"9ad31cc23cdd361a079ba989117224380daeeb072b57b3d9ed4757fc11ddd149"} Sep 16 18:52:34.921342 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:34.921016 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:52:34.938288 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:34.938236 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" podStartSLOduration=2.526987042 podStartE2EDuration="3.938219114s" totalImagesPullingTime="1.411232072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:34.938105113 +0000 UTC m=+418.060952622" watchObservedRunningTime="2026-09-16 18:52:34.938219114 +0000 UTC m=+418.061066622" Sep 16 18:52:51.906882 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:51.906852 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mzqkv" Sep 16 18:52:55.927397 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:52:55.927348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-6s47q" Sep 16 18:53:56.668076 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.668043 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-65qdg"] Sep 16 18:53:56.670143 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.670129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:56.672751 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.672727 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 18:53:56.672879 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.672730 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tf6nl\"" Sep 16 18:53:56.673445 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.673430 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 18:53:56.673688 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.673675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 18:53:56.687097 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.687076 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-65qdg"] Sep 16 18:53:56.712516 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.712496 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-x5swn"] Sep 16 18:53:56.713579 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.713562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:56.713628 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.713600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwnr\" (UniqueName: \"kubernetes.io/projected/c9a6e8d0-e3ca-4303-963e-911e95452226-kube-api-access-thwnr\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:56.714574 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.714558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:56.716821 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.716806 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-45jfk\"" Sep 16 18:53:56.716870 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.716842 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 18:53:56.725871 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.725850 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-x5swn"] Sep 16 18:53:56.814514 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.814487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645dt\" (UniqueName: \"kubernetes.io/projected/9096a269-a0b2-40a5-a4cf-94028071db62-kube-api-access-645dt\") pod \"seaweedfs-64568bcd49-x5swn\" (UID: \"9096a269-a0b2-40a5-a4cf-94028071db62\") " pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:56.814618 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.814522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:56.814618 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.814552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwnr\" (UniqueName: \"kubernetes.io/projected/c9a6e8d0-e3ca-4303-963e-911e95452226-kube-api-access-thwnr\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:56.814703 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.814612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9096a269-a0b2-40a5-a4cf-94028071db62-data\") pod \"seaweedfs-64568bcd49-x5swn\" (UID: \"9096a269-a0b2-40a5-a4cf-94028071db62\") " pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:56.814703 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:53:56.814640 2580 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 16 18:53:56.814703 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:53:56.814702 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert podName:c9a6e8d0-e3ca-4303-963e-911e95452226 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:57.314684532 +0000 UTC m=+500.437532030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert") pod "kserve-controller-manager-764fc98f48-65qdg" (UID: "c9a6e8d0-e3ca-4303-963e-911e95452226") : secret "kserve-webhook-server-cert" not found Sep 16 18:53:56.826434 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.826375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwnr\" (UniqueName: \"kubernetes.io/projected/c9a6e8d0-e3ca-4303-963e-911e95452226-kube-api-access-thwnr\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:56.915241 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.915216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645dt\" (UniqueName: \"kubernetes.io/projected/9096a269-a0b2-40a5-a4cf-94028071db62-kube-api-access-645dt\") pod \"seaweedfs-64568bcd49-x5swn\" (UID: \"9096a269-a0b2-40a5-a4cf-94028071db62\") " pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:56.915328 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.915268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9096a269-a0b2-40a5-a4cf-94028071db62-data\") pod \"seaweedfs-64568bcd49-x5swn\" (UID: \"9096a269-a0b2-40a5-a4cf-94028071db62\") " pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:56.915563 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.915549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9096a269-a0b2-40a5-a4cf-94028071db62-data\") pod \"seaweedfs-64568bcd49-x5swn\" (UID: \"9096a269-a0b2-40a5-a4cf-94028071db62\") " pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:56.924459 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:56.924420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-645dt\" (UniqueName: \"kubernetes.io/projected/9096a269-a0b2-40a5-a4cf-94028071db62-kube-api-access-645dt\") pod \"seaweedfs-64568bcd49-x5swn\" (UID: \"9096a269-a0b2-40a5-a4cf-94028071db62\") " pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:57.022654 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:57.022632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:53:57.296756 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:57.296734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-x5swn"] Sep 16 18:53:57.298761 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:53:57.298735 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9096a269_a0b2_40a5_a4cf_94028071db62.slice/crio-49709c4d3913f2874e11f05a95f20d88862dfa8a0e0ba948ccd0340aa44a7133 WatchSource:0}: Error finding container 49709c4d3913f2874e11f05a95f20d88862dfa8a0e0ba948ccd0340aa44a7133: Status 404 returned error can't find the container with id 49709c4d3913f2874e11f05a95f20d88862dfa8a0e0ba948ccd0340aa44a7133 Sep 16 18:53:57.318404 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:57.318366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:57.320765 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:57.320742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert\") pod \"kserve-controller-manager-764fc98f48-65qdg\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:57.579787 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:57.579729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:53:57.860438 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:57.860405 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-65qdg"] Sep 16 18:53:57.884989 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:53:57.884956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a6e8d0_e3ca_4303_963e_911e95452226.slice/crio-a108c449e591c7403434326f07c88f294c5c24f99d386b078babaef8dcf825a1 WatchSource:0}: Error finding container a108c449e591c7403434326f07c88f294c5c24f99d386b078babaef8dcf825a1: Status 404 returned error can't find the container with id a108c449e591c7403434326f07c88f294c5c24f99d386b078babaef8dcf825a1 Sep 16 18:53:58.257325 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:58.257206 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" event={"ID":"c9a6e8d0-e3ca-4303-963e-911e95452226","Type":"ContainerStarted","Data":"a108c449e591c7403434326f07c88f294c5c24f99d386b078babaef8dcf825a1"} Sep 16 18:53:58.258626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:53:58.258601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-x5swn" event={"ID":"9096a269-a0b2-40a5-a4cf-94028071db62","Type":"ContainerStarted","Data":"49709c4d3913f2874e11f05a95f20d88862dfa8a0e0ba948ccd0340aa44a7133"} Sep 16 18:54:01.275916 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:01.275826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" event={"ID":"c9a6e8d0-e3ca-4303-963e-911e95452226","Type":"ContainerStarted","Data":"76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af"} Sep 16 18:54:01.276310 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:01.275926 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:54:01.277433 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:01.277409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-x5swn" event={"ID":"9096a269-a0b2-40a5-a4cf-94028071db62","Type":"ContainerStarted","Data":"d92f90128b7c3a25e81ae58b2bab0bdfd9552990e0c0b7a0b35d86086b67fb69"} Sep 16 18:54:01.277518 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:01.277474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:54:01.294858 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:01.294814 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" podStartSLOduration=2.125676917 podStartE2EDuration="5.294800213s" totalImagesPullingTime="3.169123296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:01.293572696 +0000 UTC m=+504.416420203" watchObservedRunningTime="2026-09-16 18:54:01.294800213 +0000 UTC m=+504.417647720" Sep 16 18:54:01.313331 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:01.313296 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-x5swn" podStartSLOduration=1.528533641 podStartE2EDuration="5.313278943s" totalImagesPullingTime="3.784745302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:01.313064733 +0000 UTC m=+504.435912241" watchObservedRunningTime="2026-09-16 18:54:01.313278943 +0000 UTC m=+504.436126438" Sep 16 18:54:07.283055 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:07.283025 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-x5swn" Sep 16 18:54:32.286166 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:32.286098 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:54:39.849309 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:39.849269 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-65qdg"] Sep 16 18:54:39.849899 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:39.849514 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" podUID="c9a6e8d0-e3ca-4303-963e-911e95452226" containerName="manager" containerID="cri-o://76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af" gracePeriod=10 Sep 16 18:54:39.890810 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:39.890785 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-gz6dk"] Sep 16 18:54:39.892766 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:39.892752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:39.903745 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:39.903721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-gz6dk"] Sep 16 18:54:40.023896 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.023863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/412944c9-5047-4e05-b933-447a7ddeefff-cert\") pod \"kserve-controller-manager-764fc98f48-gz6dk\" (UID: \"412944c9-5047-4e05-b933-447a7ddeefff\") " pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.024055 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.024018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jcj\" (UniqueName: \"kubernetes.io/projected/412944c9-5047-4e05-b933-447a7ddeefff-kube-api-access-67jcj\") pod \"kserve-controller-manager-764fc98f48-gz6dk\" (UID: \"412944c9-5047-4e05-b933-447a7ddeefff\") " pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.089336 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.089316 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:54:40.124563 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.124504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/412944c9-5047-4e05-b933-447a7ddeefff-cert\") pod \"kserve-controller-manager-764fc98f48-gz6dk\" (UID: \"412944c9-5047-4e05-b933-447a7ddeefff\") " pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.124690 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.124619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jcj\" (UniqueName: \"kubernetes.io/projected/412944c9-5047-4e05-b933-447a7ddeefff-kube-api-access-67jcj\") pod \"kserve-controller-manager-764fc98f48-gz6dk\" (UID: \"412944c9-5047-4e05-b933-447a7ddeefff\") " pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.126887 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.126863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/412944c9-5047-4e05-b933-447a7ddeefff-cert\") pod \"kserve-controller-manager-764fc98f48-gz6dk\" (UID: \"412944c9-5047-4e05-b933-447a7ddeefff\") " pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.133777 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.133752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jcj\" (UniqueName: \"kubernetes.io/projected/412944c9-5047-4e05-b933-447a7ddeefff-kube-api-access-67jcj\") pod \"kserve-controller-manager-764fc98f48-gz6dk\" (UID: \"412944c9-5047-4e05-b933-447a7ddeefff\") " pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.225145 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.225121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thwnr\" (UniqueName: \"kubernetes.io/projected/c9a6e8d0-e3ca-4303-963e-911e95452226-kube-api-access-thwnr\") pod \"c9a6e8d0-e3ca-4303-963e-911e95452226\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " Sep 16 18:54:40.225238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.225174 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert\") pod \"c9a6e8d0-e3ca-4303-963e-911e95452226\" (UID: \"c9a6e8d0-e3ca-4303-963e-911e95452226\") " Sep 16 18:54:40.227210 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.227187 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert" (OuterVolumeSpecName: "cert") pod "c9a6e8d0-e3ca-4303-963e-911e95452226" (UID: "c9a6e8d0-e3ca-4303-963e-911e95452226"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:54:40.227292 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.227250 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a6e8d0-e3ca-4303-963e-911e95452226-kube-api-access-thwnr" (OuterVolumeSpecName: "kube-api-access-thwnr") pod "c9a6e8d0-e3ca-4303-963e-911e95452226" (UID: "c9a6e8d0-e3ca-4303-963e-911e95452226"). InnerVolumeSpecName "kube-api-access-thwnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:54:40.241062 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.241044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:40.326395 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.326350 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thwnr\" (UniqueName: \"kubernetes.io/projected/c9a6e8d0-e3ca-4303-963e-911e95452226-kube-api-access-thwnr\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:54:40.326395 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.326389 2580 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9a6e8d0-e3ca-4303-963e-911e95452226-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:54:40.439874 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.439814 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9a6e8d0-e3ca-4303-963e-911e95452226" containerID="76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af" exitCode=0 Sep 16 18:54:40.439874 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.439846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" event={"ID":"c9a6e8d0-e3ca-4303-963e-911e95452226","Type":"ContainerDied","Data":"76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af"} Sep 16 18:54:40.439874 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.439858 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" Sep 16 18:54:40.440078 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.439881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-65qdg" event={"ID":"c9a6e8d0-e3ca-4303-963e-911e95452226","Type":"ContainerDied","Data":"a108c449e591c7403434326f07c88f294c5c24f99d386b078babaef8dcf825a1"} Sep 16 18:54:40.440078 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.439901 2580 scope.go:117] "RemoveContainer" containerID="76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af" Sep 16 18:54:40.447944 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.447928 2580 scope.go:117] "RemoveContainer" containerID="76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af" Sep 16 18:54:40.448205 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:54:40.448189 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af\": container with ID starting with 76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af not found: ID does not exist" containerID="76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af" Sep 16 18:54:40.448264 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.448218 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af"} err="failed to get container status \"76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af\": rpc error: code = NotFound desc = could not find container \"76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af\": container with ID starting with 76c22b83554b2779dd7799b383f67a364b8a5df407497ca7b41676f5f6cca0af not found: ID does not exist" Sep 16 18:54:40.460218 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.460193 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-65qdg"] Sep 16 18:54:40.463655 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.463638 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-65qdg"] Sep 16 18:54:40.506989 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:40.506906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-gz6dk"] Sep 16 18:54:40.509296 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:54:40.509277 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412944c9_5047_4e05_b933_447a7ddeefff.slice/crio-0dec04b5445c674f448fc7dbba527f5e7955a94a370e21554e03abd16fb3636b WatchSource:0}: Error finding container 0dec04b5445c674f448fc7dbba527f5e7955a94a370e21554e03abd16fb3636b: Status 404 returned error can't find the container with id 0dec04b5445c674f448fc7dbba527f5e7955a94a370e21554e03abd16fb3636b Sep 16 18:54:41.401676 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:41.401649 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a6e8d0-e3ca-4303-963e-911e95452226" path="/var/lib/kubelet/pods/c9a6e8d0-e3ca-4303-963e-911e95452226/volumes" Sep 16 18:54:41.445581 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:41.445552 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" event={"ID":"412944c9-5047-4e05-b933-447a7ddeefff","Type":"ContainerStarted","Data":"82699d17ca08bebbdb7229ab92f74f979f86717f7d48310fcadd9d75ba7720fc"} Sep 16 18:54:41.445723 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:41.445596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" event={"ID":"412944c9-5047-4e05-b933-447a7ddeefff","Type":"ContainerStarted","Data":"0dec04b5445c674f448fc7dbba527f5e7955a94a370e21554e03abd16fb3636b"} Sep 16 18:54:41.445723 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:41.445637 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:54:41.462720 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:54:41.462546 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" podStartSLOduration=2.053898084 podStartE2EDuration="2.462528668s" totalImagesPullingTime="408.630584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:41.462257932 +0000 UTC m=+544.585105437" watchObservedRunningTime="2026-09-16 18:54:41.462528668 +0000 UTC m=+544.585376155" Sep 16 18:55:12.454322 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:12.454289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-764fc98f48-gz6dk" Sep 16 18:55:13.932344 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.932314 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-4j46p"] Sep 16 18:55:13.932695 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.932659 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a6e8d0-e3ca-4303-963e-911e95452226" containerName="manager" Sep 16 18:55:13.932695 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.932671 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a6e8d0-e3ca-4303-963e-911e95452226" containerName="manager" Sep 16 18:55:13.932769 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.932734 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a6e8d0-e3ca-4303-963e-911e95452226" containerName="manager" Sep 16 18:55:13.935891 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.935870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:13.938121 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.938097 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-cdls7\"" Sep 16 18:55:13.945773 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.945754 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 16 18:55:13.977278 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.977245 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-w8hq8"] Sep 16 18:55:13.980437 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.980420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:13.982795 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.982774 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Sep 16 18:55:13.982795 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.982791 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Sep 16 18:55:13.983163 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.983149 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-vjkwg\"" Sep 16 18:55:13.990652 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:13.990631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-w8hq8"] Sep 16 18:55:14.012866 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.012825 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-4j46p"] Sep 16 18:55:14.067529 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.067509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d95fff-1233-46ed-8a5a-842ff5ba8112-tls-certs\") pod \"model-serving-api-7b954ffc95-4j46p\" (UID: \"e5d95fff-1233-46ed-8a5a-842ff5ba8112\") " pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.067626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.067541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4jg\" (UniqueName: \"kubernetes.io/projected/e5d95fff-1233-46ed-8a5a-842ff5ba8112-kube-api-access-qt4jg\") pod \"model-serving-api-7b954ffc95-4j46p\" (UID: \"e5d95fff-1233-46ed-8a5a-842ff5ba8112\") " pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.168904 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.168880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d95fff-1233-46ed-8a5a-842ff5ba8112-tls-certs\") pod \"model-serving-api-7b954ffc95-4j46p\" (UID: \"e5d95fff-1233-46ed-8a5a-842ff5ba8112\") " pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.169000 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.168914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4jg\" (UniqueName: \"kubernetes.io/projected/e5d95fff-1233-46ed-8a5a-842ff5ba8112-kube-api-access-qt4jg\") pod \"model-serving-api-7b954ffc95-4j46p\" (UID: \"e5d95fff-1233-46ed-8a5a-842ff5ba8112\") " pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.169000 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.168947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-cert\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.169000 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.168972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-metrics-certs\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.169142 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.169003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56dp\" (UniqueName: \"kubernetes.io/projected/89a18619-da35-4341-8e1a-0dd02d8796ae-kube-api-access-k56dp\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.171416 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.171372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d95fff-1233-46ed-8a5a-842ff5ba8112-tls-certs\") pod \"model-serving-api-7b954ffc95-4j46p\" (UID: \"e5d95fff-1233-46ed-8a5a-842ff5ba8112\") " pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.177826 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.177808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4jg\" (UniqueName: \"kubernetes.io/projected/e5d95fff-1233-46ed-8a5a-842ff5ba8112-kube-api-access-qt4jg\") pod \"model-serving-api-7b954ffc95-4j46p\" (UID: \"e5d95fff-1233-46ed-8a5a-842ff5ba8112\") " pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.247719 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.247655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:14.269655 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.269631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-cert\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.269747 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.269669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-metrics-certs\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.269747 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.269704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56dp\" (UniqueName: \"kubernetes.io/projected/89a18619-da35-4341-8e1a-0dd02d8796ae-kube-api-access-k56dp\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.270013 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:55:14.269776 2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 16 18:55:14.270013 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:55:14.269831 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-cert podName:89a18619-da35-4341-8e1a-0dd02d8796ae nodeName:}" failed. No retries permitted until 2026-09-16 18:55:14.769816521 +0000 UTC m=+577.892664008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-cert") pod "odh-model-controller-7b966d7984-w8hq8" (UID: "89a18619-da35-4341-8e1a-0dd02d8796ae") : secret "odh-model-controller-webhook-cert" not found Sep 16 18:55:14.272068 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.272049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-metrics-certs\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.278767 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.278747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56dp\" (UniqueName: \"kubernetes.io/projected/89a18619-da35-4341-8e1a-0dd02d8796ae-kube-api-access-k56dp\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.524934 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.524908 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-4j46p"] Sep 16 18:55:14.526895 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:55:14.526867 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d95fff_1233_46ed_8a5a_842ff5ba8112.slice/crio-2201f7e5824a45f9af56d41f515a084b478acdf348b4aa424affbc57190e1ce1 WatchSource:0}: Error finding container 2201f7e5824a45f9af56d41f515a084b478acdf348b4aa424affbc57190e1ce1: Status 404 returned error can't find the container with id 2201f7e5824a45f9af56d41f515a084b478acdf348b4aa424affbc57190e1ce1 Sep 16 18:55:14.580212 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.580187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-4j46p" event={"ID":"e5d95fff-1233-46ed-8a5a-842ff5ba8112","Type":"ContainerStarted","Data":"2201f7e5824a45f9af56d41f515a084b478acdf348b4aa424affbc57190e1ce1"} Sep 16 18:55:14.773782 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.773752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-cert\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.776242 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.776185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89a18619-da35-4341-8e1a-0dd02d8796ae-cert\") pod \"odh-model-controller-7b966d7984-w8hq8\" (UID: \"89a18619-da35-4341-8e1a-0dd02d8796ae\") " pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:14.890544 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:14.890516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:15.164928 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:15.164895 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-w8hq8"] Sep 16 18:55:15.167846 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:55:15.167818 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a18619_da35_4341_8e1a_0dd02d8796ae.slice/crio-741018abdb11d3e7f7221b6b9b2738982816063c0ed7109054150d913b93ffe7 WatchSource:0}: Error finding container 741018abdb11d3e7f7221b6b9b2738982816063c0ed7109054150d913b93ffe7: Status 404 returned error can't find the container with id 741018abdb11d3e7f7221b6b9b2738982816063c0ed7109054150d913b93ffe7 Sep 16 18:55:15.587274 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:15.587241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-w8hq8" event={"ID":"89a18619-da35-4341-8e1a-0dd02d8796ae","Type":"ContainerStarted","Data":"741018abdb11d3e7f7221b6b9b2738982816063c0ed7109054150d913b93ffe7"} Sep 16 18:55:18.603443 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:18.603410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-4j46p" event={"ID":"e5d95fff-1233-46ed-8a5a-842ff5ba8112","Type":"ContainerStarted","Data":"3f3496baa413e7fcc98b6d0950abcc135ca9353ebdc12fe313416d4dff7ae28b"} Sep 16 18:55:18.603917 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:18.603519 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:18.604947 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:18.604923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-w8hq8" event={"ID":"89a18619-da35-4341-8e1a-0dd02d8796ae","Type":"ContainerStarted","Data":"81dd202979e07d83a8180fb072a7e014c863aaae0ba889680f25787785758b6e"} Sep 16 18:55:18.605077 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:18.604973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:18.626859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:18.626807 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-4j46p" podStartSLOduration=1.982874617 podStartE2EDuration="5.626792282s" totalImagesPullingTime="3.643917665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:18.62676055 +0000 UTC m=+581.749608056" watchObservedRunningTime="2026-09-16 18:55:18.626792282 +0000 UTC m=+581.749639791" Sep 16 18:55:18.647838 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:18.647798 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-w8hq8" podStartSLOduration=2.599785831 podStartE2EDuration="5.647783559s" totalImagesPullingTime="3.047997728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:18.646689498 +0000 UTC m=+581.769537004" watchObservedRunningTime="2026-09-16 18:55:18.647783559 +0000 UTC m=+581.770631090" Sep 16 18:55:29.612525 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:29.612498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-w8hq8" Sep 16 18:55:29.614346 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:29.614326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-4j46p" Sep 16 18:55:37.297457 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:37.297430 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:55:37.305461 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:55:37.305437 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 18:57:01.680658 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.680626 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl"] Sep 16 18:57:01.684099 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.684078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.687933 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.687911 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:57:01.688373 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.688358 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Sep 16 18:57:01.688749 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.688729 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:57:01.688859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.688774 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\"" Sep 16 18:57:01.688859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.688787 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:57:01.688859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.688800 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:57:01.688859 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.688793 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Sep 16 18:57:01.699653 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.699632 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl"] Sep 16 18:57:01.785668 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.785639 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv"] Sep 16 18:57:01.789100 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.789084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:01.791309 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.791286 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Sep 16 18:57:01.803418 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.803360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv"] Sep 16 18:57:01.825119 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.825091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294bf32f-c3a5-4130-a342-3be80ab672dd-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.825253 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.825130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.825253 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.825228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.825363 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.825269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.825363 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.825342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2l9t\" (UniqueName: \"kubernetes.io/projected/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-api-access-l2l9t\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.926796 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l9t\" (UniqueName: \"kubernetes.io/projected/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-api-access-l2l9t\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.926960 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv58b\" (UniqueName: \"kubernetes.io/projected/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-api-access-xv58b\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:01.926960 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:01.926960 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98926bef-23e1-4c62-ad77-c4edc05bf0bb-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:01.926960 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294bf32f-c3a5-4130-a342-3be80ab672dd-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.927131 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.927131 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.926988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:01.927131 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.927010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:01.927131 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.927085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.927131 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.927128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.927293 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:57:01.927253 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found Sep 16 18:57:01.927331 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:57:01.927315 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls podName:294bf32f-c3a5-4130-a342-3be80ab672dd nodeName:}" failed. No retries permitted until 2026-09-16 18:57:02.427294823 +0000 UTC m=+685.550142313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls") pod "isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" (UID: "294bf32f-c3a5-4130-a342-3be80ab672dd") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found Sep 16 18:57:01.927331 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.927316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294bf32f-c3a5-4130-a342-3be80ab672dd-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.927678 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.927658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.927762 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.927746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:01.935635 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:01.935572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l9t\" (UniqueName: \"kubernetes.io/projected/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-api-access-l2l9t\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:02.028351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.028313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv58b\" (UniqueName: \"kubernetes.io/projected/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-api-access-xv58b\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.028351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.028353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.028626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.028410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98926bef-23e1-4c62-ad77-c4edc05bf0bb-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.028626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.028449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.028626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.028473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.028909 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.028885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.029123 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.029099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.029201 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.029163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.030973 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.030952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98926bef-23e1-4c62-ad77-c4edc05bf0bb-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.037716 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.037693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv58b\" (UniqueName: \"kubernetes.io/projected/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-api-access-xv58b\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-qblrv\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.099302 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.099278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:02.378085 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.378062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv"] Sep 16 18:57:02.380252 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:57:02.380220 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98926bef_23e1_4c62_ad77_c4edc05bf0bb.slice/crio-2f02a753ce367a62fdd21ccfae8c8f673051b4e611969c02f6b6cc1c82764ce8 WatchSource:0}: Error finding container 2f02a753ce367a62fdd21ccfae8c8f673051b4e611969c02f6b6cc1c82764ce8: Status 404 returned error can't find the container with id 2f02a753ce367a62fdd21ccfae8c8f673051b4e611969c02f6b6cc1c82764ce8 Sep 16 18:57:02.432208 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.432181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:02.434523 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.434506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-97ljl\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:02.594644 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.594607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:02.867075 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:02.867051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl"] Sep 16 18:57:02.868812 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:57:02.868787 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294bf32f_c3a5_4130_a342_3be80ab672dd.slice/crio-f58168b21769b5133d699bf6cef0b275adcd6162970c89027c53a5b6550c0031 WatchSource:0}: Error finding container f58168b21769b5133d699bf6cef0b275adcd6162970c89027c53a5b6550c0031: Status 404 returned error can't find the container with id f58168b21769b5133d699bf6cef0b275adcd6162970c89027c53a5b6550c0031 Sep 16 18:57:03.043537 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:03.043512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerStarted","Data":"f58168b21769b5133d699bf6cef0b275adcd6162970c89027c53a5b6550c0031"} Sep 16 18:57:03.044619 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:03.044596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerStarted","Data":"2f02a753ce367a62fdd21ccfae8c8f673051b4e611969c02f6b6cc1c82764ce8"} Sep 16 18:57:07.068674 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:07.068636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerStarted","Data":"a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827"} Sep 16 18:57:07.070323 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:07.070299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerStarted","Data":"5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7"} Sep 16 18:57:10.085074 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:10.085048 2580 generic.go:356] "Generic (PLEG): container finished" podID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerID="a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827" exitCode=0 Sep 16 18:57:10.085418 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:10.085133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerDied","Data":"a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827"} Sep 16 18:57:10.086775 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:10.086751 2580 generic.go:356] "Generic (PLEG): container finished" podID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerID="5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7" exitCode=0 Sep 16 18:57:10.086888 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:10.086781 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerDied","Data":"5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7"} Sep 16 18:57:27.514849 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:27.514830 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:57:28.247335 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:28.247249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerStarted","Data":"ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806"} Sep 16 18:57:29.255102 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:29.255065 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerStarted","Data":"35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d"} Sep 16 18:57:30.262394 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.262299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerStarted","Data":"dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f"} Sep 16 18:57:30.262828 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.262547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:30.262828 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.262666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:30.263756 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.263722 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:57:30.264713 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.264689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerStarted","Data":"411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0"} Sep 16 18:57:30.264846 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.264827 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:30.264903 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.264881 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:30.265883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.265849 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:57:30.286846 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.286802 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podStartSLOduration=2.659134371 podStartE2EDuration="29.28679116s" totalImagesPullingTime="22.77469059s" totalInitContainerRuntime="3.852966199s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:30.286208532 +0000 UTC m=+713.409056038" watchObservedRunningTime="2026-09-16 18:57:30.28679116 +0000 UTC m=+713.409638665" Sep 16 18:57:30.316679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:30.316638 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podStartSLOduration=2.132077338 podStartE2EDuration="29.31662414s" totalImagesPullingTime="23.314119417s" totalInitContainerRuntime="3.870427385s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:30.316070594 +0000 UTC m=+713.438918105" watchObservedRunningTime="2026-09-16 18:57:30.31662414 +0000 UTC m=+713.439471646" Sep 16 18:57:31.270079 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:31.270034 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:57:31.270498 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:31.270162 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:57:36.274471 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:36.274439 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:57:36.275033 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:36.274528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:57:36.275033 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:36.275012 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:57:36.275172 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:36.275100 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:57:46.275597 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:46.275561 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:57:46.275597 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:46.275579 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:57:56.274999 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:56.274960 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:57:56.275368 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:57:56.274961 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:58:06.275644 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:06.275608 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:58:06.275644 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:06.275622 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:58:16.275441 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:16.275404 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:58:16.275441 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:16.275418 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:58:26.275142 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:26.275098 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:58:26.275547 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:26.275139 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:58:36.275470 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:36.275422 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 16 18:58:36.275876 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:36.275422 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 16 18:58:41.401494 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:41.401461 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:58:46.276178 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:46.276151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 18:58:51.414301 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:51.414268 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl"] Sep 16 18:58:51.414883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:51.414505 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" containerID="cri-o://ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806" gracePeriod=30 Sep 16 18:58:51.414883 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:51.414559 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kube-rbac-proxy" containerID="cri-o://dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f" gracePeriod=30 Sep 16 18:58:51.633393 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:51.633347 2580 generic.go:356] "Generic (PLEG): container finished" podID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerID="dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f" exitCode=2 Sep 16 18:58:51.633532 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:51.633422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerDied","Data":"dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f"} Sep 16 18:58:55.354719 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.354686 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:58:55.433019 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.432987 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294bf32f-c3a5-4130-a342-3be80ab672dd-kserve-provision-location\") pod \"294bf32f-c3a5-4130-a342-3be80ab672dd\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " Sep 16 18:58:55.433160 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433029 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-cabundle-cert\") pod \"294bf32f-c3a5-4130-a342-3be80ab672dd\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " Sep 16 18:58:55.433160 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433050 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls\") pod \"294bf32f-c3a5-4130-a342-3be80ab672dd\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " Sep 16 18:58:55.433160 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2l9t\" (UniqueName: \"kubernetes.io/projected/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-api-access-l2l9t\") pod \"294bf32f-c3a5-4130-a342-3be80ab672dd\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " Sep 16 18:58:55.433160 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433082 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-rbac-proxy-sar-config\") pod \"294bf32f-c3a5-4130-a342-3be80ab672dd\" (UID: \"294bf32f-c3a5-4130-a342-3be80ab672dd\") " Sep 16 18:58:55.433428 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433348 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "294bf32f-c3a5-4130-a342-3be80ab672dd" (UID: "294bf32f-c3a5-4130-a342-3be80ab672dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:58:55.433428 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433342 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294bf32f-c3a5-4130-a342-3be80ab672dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "294bf32f-c3a5-4130-a342-3be80ab672dd" (UID: "294bf32f-c3a5-4130-a342-3be80ab672dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:58:55.433591 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.433511 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "294bf32f-c3a5-4130-a342-3be80ab672dd" (UID: "294bf32f-c3a5-4130-a342-3be80ab672dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:58:55.435364 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.435338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-api-access-l2l9t" (OuterVolumeSpecName: "kube-api-access-l2l9t") pod "294bf32f-c3a5-4130-a342-3be80ab672dd" (UID: "294bf32f-c3a5-4130-a342-3be80ab672dd"). InnerVolumeSpecName "kube-api-access-l2l9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:58:55.435447 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.435430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "294bf32f-c3a5-4130-a342-3be80ab672dd" (UID: "294bf32f-c3a5-4130-a342-3be80ab672dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:58:55.533775 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.533719 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294bf32f-c3a5-4130-a342-3be80ab672dd-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:58:55.533775 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.533740 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:58:55.533775 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.533754 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294bf32f-c3a5-4130-a342-3be80ab672dd-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:58:55.533775 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.533766 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2l9t\" (UniqueName: \"kubernetes.io/projected/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-api-access-l2l9t\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:58:55.533952 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.533778 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294bf32f-c3a5-4130-a342-3be80ab672dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:58:55.652624 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.652593 2580 generic.go:356] "Generic (PLEG): container finished" podID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerID="ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806" exitCode=0 Sep 16 18:58:55.652771 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.652661 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" Sep 16 18:58:55.652771 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.652664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerDied","Data":"ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806"} Sep 16 18:58:55.652885 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.652789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl" event={"ID":"294bf32f-c3a5-4130-a342-3be80ab672dd","Type":"ContainerDied","Data":"f58168b21769b5133d699bf6cef0b275adcd6162970c89027c53a5b6550c0031"} Sep 16 18:58:55.652885 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.652818 2580 scope.go:117] "RemoveContainer" containerID="dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f" Sep 16 18:58:55.661878 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.661863 2580 scope.go:117] "RemoveContainer" containerID="ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806" Sep 16 18:58:55.669151 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.669135 2580 scope.go:117] "RemoveContainer" containerID="a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827" Sep 16 18:58:55.676853 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.676831 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl"] Sep 16 18:58:55.678360 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.678340 2580 scope.go:117] "RemoveContainer" containerID="dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f" Sep 16 18:58:55.678723 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:58:55.678704 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f\": container with ID starting with dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f not found: ID does not exist" containerID="dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f" Sep 16 18:58:55.678800 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.678731 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f"} err="failed to get container status \"dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f\": rpc error: code = NotFound desc = could not find container \"dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f\": container with ID starting with dc9658dac2737bd55d29aef754543593b781d336659af46f2ae7df0598ba2a9f not found: ID does not exist" Sep 16 18:58:55.678800 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.678747 2580 scope.go:117] "RemoveContainer" containerID="ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806" Sep 16 18:58:55.679025 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:58:55.679004 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806\": container with ID starting with ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806 not found: ID does not exist" containerID="ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806" Sep 16 18:58:55.679075 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.679031 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806"} err="failed to get container status \"ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806\": rpc error: code = NotFound desc = could not find container \"ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806\": container with ID starting with ec07c0f7bf908f1707b80d49fb3e8f36bee16a048835d25491a96340b5278806 not found: ID does not exist" Sep 16 18:58:55.679075 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.679045 2580 scope.go:117] "RemoveContainer" containerID="a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827" Sep 16 18:58:55.679286 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:58:55.679269 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827\": container with ID starting with a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827 not found: ID does not exist" containerID="a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827" Sep 16 18:58:55.679329 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.679292 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827"} err="failed to get container status \"a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827\": rpc error: code = NotFound desc = could not find container \"a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827\": container with ID starting with a19831d18995b996f200731d90e4ad84243110c5fc2d085aaa321638464e8827 not found: ID does not exist" Sep 16 18:58:55.679732 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:55.679714 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-97ljl"] Sep 16 18:58:56.208774 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.208739 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-878585977-6vxkv"] Sep 16 18:58:56.209067 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209056 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="storage-initializer" Sep 16 18:58:56.209110 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209069 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="storage-initializer" Sep 16 18:58:56.209110 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209078 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kube-rbac-proxy" Sep 16 18:58:56.209110 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209083 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kube-rbac-proxy" Sep 16 18:58:56.209110 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209098 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" Sep 16 18:58:56.209110 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209103 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" Sep 16 18:58:56.209259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209159 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kserve-container" Sep 16 18:58:56.209259 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.209166 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" containerName="kube-rbac-proxy" Sep 16 18:58:56.213601 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.213580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.224497 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.224475 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-878585977-6vxkv"] Sep 16 18:58:56.339089 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-serving-cert\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.339238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-trusted-ca-bundle\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.339238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-config\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.339238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbgn\" (UniqueName: \"kubernetes.io/projected/f1ce76f0-1313-40aa-ba05-2bba021a42a3-kube-api-access-wlbgn\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.339238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-oauth-serving-cert\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.339238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-oauth-config\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.339238 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.339235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-service-ca\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440201 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbgn\" (UniqueName: \"kubernetes.io/projected/f1ce76f0-1313-40aa-ba05-2bba021a42a3-kube-api-access-wlbgn\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-oauth-serving-cert\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-oauth-config\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-service-ca\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-serving-cert\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-trusted-ca-bundle\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.440679 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.440605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-config\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.441495 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.441158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-service-ca\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.441666 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.441636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-trusted-ca-bundle\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.441825 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.441773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-config\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.442323 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.442144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1ce76f0-1313-40aa-ba05-2bba021a42a3-oauth-serving-cert\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.443129 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.443109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-oauth-config\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.443427 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.443409 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1ce76f0-1313-40aa-ba05-2bba021a42a3-console-serving-cert\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.451373 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.451351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbgn\" (UniqueName: \"kubernetes.io/projected/f1ce76f0-1313-40aa-ba05-2bba021a42a3-kube-api-access-wlbgn\") pod \"console-878585977-6vxkv\" (UID: \"f1ce76f0-1313-40aa-ba05-2bba021a42a3\") " pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.525247 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.525179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-878585977-6vxkv" Sep 16 18:58:56.625730 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.625703 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq"] Sep 16 18:58:56.630205 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.630188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.632498 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.632474 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-1ba12-predictor-serving-cert\"" Sep 16 18:58:56.632595 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.632523 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-1ba12-kube-rbac-proxy-sar-config\"" Sep 16 18:58:56.642113 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.642087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq"] Sep 16 18:58:56.743536 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.743500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.743683 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.743571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.743683 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.743600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfl68\" (UniqueName: \"kubernetes.io/projected/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-api-access-jfl68\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.743683 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.743628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.743791 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.743700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.813148 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.813047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-878585977-6vxkv"] Sep 16 18:58:56.815768 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:58:56.815741 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ce76f0_1313_40aa_ba05_2bba021a42a3.slice/crio-e9855c75a2868c41681a9da5d26867960af8956ad41f21bc808916859fafcc23 WatchSource:0}: Error finding container e9855c75a2868c41681a9da5d26867960af8956ad41f21bc808916859fafcc23: Status 404 returned error can't find the container with id e9855c75a2868c41681a9da5d26867960af8956ad41f21bc808916859fafcc23 Sep 16 18:58:56.845022 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.844995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.845120 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.845051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.845186 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:58:56.845149 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-serving-cert: secret "isvc-sklearn-graph-raw-1ba12-predictor-serving-cert" not found Sep 16 18:58:56.845186 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.845158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfl68\" (UniqueName: \"kubernetes.io/projected/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-api-access-jfl68\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.845297 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:58:56.845199 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls podName:3624fc6d-dc56-4b96-b4b2-0109e7940aa5 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:57.345185687 +0000 UTC m=+800.468033171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls") pod "isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" (UID: "3624fc6d-dc56-4b96-b4b2-0109e7940aa5") : secret "isvc-sklearn-graph-raw-1ba12-predictor-serving-cert" not found Sep 16 18:58:56.845297 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.845212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.845297 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.845240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.845555 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.845531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.845892 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.845867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.846065 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.846049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:56.855984 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:56.855955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfl68\" (UniqueName: \"kubernetes.io/projected/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-api-access-jfl68\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:57.348299 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.348270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:57.350676 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.350646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls\") pod \"isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:57.401760 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.401735 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294bf32f-c3a5-4130-a342-3be80ab672dd" path="/var/lib/kubelet/pods/294bf32f-c3a5-4130-a342-3be80ab672dd/volumes" Sep 16 18:58:57.541497 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.541477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:58:57.664630 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.664558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-878585977-6vxkv" event={"ID":"f1ce76f0-1313-40aa-ba05-2bba021a42a3","Type":"ContainerStarted","Data":"3339ad1b3cb2f89e5dfc69e5ab10b1321cd07a67dc11cd2f37984c5e707bf631"} Sep 16 18:58:57.664630 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.664612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-878585977-6vxkv" event={"ID":"f1ce76f0-1313-40aa-ba05-2bba021a42a3","Type":"ContainerStarted","Data":"e9855c75a2868c41681a9da5d26867960af8956ad41f21bc808916859fafcc23"} Sep 16 18:58:57.688932 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.688315 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-878585977-6vxkv" podStartSLOduration=1.688296037 podStartE2EDuration="1.688296037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:57.687698289 +0000 UTC m=+800.810545800" watchObservedRunningTime="2026-09-16 18:58:57.688296037 +0000 UTC m=+800.811143544" Sep 16 18:58:57.820626 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:57.820606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq"] Sep 16 18:58:57.822420 ip-10-0-128-157 kubenswrapper[2580]: W0916 18:58:57.822396 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3624fc6d_dc56_4b96_b4b2_0109e7940aa5.slice/crio-47377aabf2dce18f35eade2f8b1d6a3d9c878edb81468d0704fb4a6304243f4d WatchSource:0}: Error finding container 47377aabf2dce18f35eade2f8b1d6a3d9c878edb81468d0704fb4a6304243f4d: Status 404 returned error can't find the container with id 47377aabf2dce18f35eade2f8b1d6a3d9c878edb81468d0704fb4a6304243f4d Sep 16 18:58:58.670037 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:58.669955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerStarted","Data":"bf677e4710a19e377a1e4556a87ef96738cccb02b964de5984fc583f731a14a2"} Sep 16 18:58:58.670414 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:58:58.670061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerStarted","Data":"47377aabf2dce18f35eade2f8b1d6a3d9c878edb81468d0704fb4a6304243f4d"} Sep 16 18:59:02.687468 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:02.687437 2580 generic.go:356] "Generic (PLEG): container finished" podID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerID="bf677e4710a19e377a1e4556a87ef96738cccb02b964de5984fc583f731a14a2" exitCode=0 Sep 16 18:59:02.687828 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:02.687517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerDied","Data":"bf677e4710a19e377a1e4556a87ef96738cccb02b964de5984fc583f731a14a2"} Sep 16 18:59:03.699712 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:03.699680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerStarted","Data":"e1b1992277623da9d656b713428002cb30d8066f27ca72bd56cf7db0b9dec0fd"} Sep 16 18:59:03.700104 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:03.699769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerStarted","Data":"bf66445d34149264f058a492ea7188aaf45848a2291435dbf8d9e4aa1c50e3c0"} Sep 16 18:59:03.700104 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:03.699970 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:59:03.721628 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:03.721576 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podStartSLOduration=3.777007584 podStartE2EDuration="7.721563331s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.944555747s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:03.719496586 +0000 UTC m=+806.842344092" watchObservedRunningTime="2026-09-16 18:59:03.721563331 +0000 UTC m=+806.844410836" Sep 16 18:59:04.704237 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:04.704200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:59:04.705457 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:04.705433 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 18:59:05.708404 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:05.708338 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 18:59:06.525826 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:06.525792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-878585977-6vxkv" Sep 16 18:59:06.525985 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:06.525849 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-878585977-6vxkv" Sep 16 18:59:06.530605 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:06.530582 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-878585977-6vxkv" Sep 16 18:59:06.716441 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:06.716418 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-878585977-6vxkv" Sep 16 18:59:06.777620 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:06.777554 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bd87c846-78cvj"] Sep 16 18:59:10.713369 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:10.713340 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 18:59:10.713875 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:10.713850 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 18:59:20.714807 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:20.714763 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 18:59:30.714578 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:30.714536 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 18:59:31.797397 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:31.797336 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55bd87c846-78cvj" podUID="4f20be76-079d-4772-8c17-50a2012848bb" containerName="console" containerID="cri-o://5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a" gracePeriod=15 Sep 16 18:59:32.039343 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.039321 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:59:32.112954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.112890 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-oauth-config\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.112954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.112927 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-oauth-serving-cert\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.112954 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.112946 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-trusted-ca-bundle\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.113193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.112970 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-service-ca\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.113193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-serving-cert\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.113193 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113171 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsr4k\" (UniqueName: \"kubernetes.io/projected/4f20be76-079d-4772-8c17-50a2012848bb-kube-api-access-gsr4k\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.113351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113195 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-console-config\") pod \"4f20be76-079d-4772-8c17-50a2012848bb\" (UID: \"4f20be76-079d-4772-8c17-50a2012848bb\") " Sep 16 18:59:32.113351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113250 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:32.113351 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113333 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:32.113502 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113362 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:32.113658 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113639 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-oauth-serving-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.113732 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113653 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-console-config" (OuterVolumeSpecName: "console-config") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:32.113732 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113665 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-trusted-ca-bundle\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.113732 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.113678 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-service-ca\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.115246 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.115209 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:32.115336 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.115309 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f20be76-079d-4772-8c17-50a2012848bb-kube-api-access-gsr4k" (OuterVolumeSpecName: "kube-api-access-gsr4k") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "kube-api-access-gsr4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:59:32.115573 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.115552 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4f20be76-079d-4772-8c17-50a2012848bb" (UID: "4f20be76-079d-4772-8c17-50a2012848bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:32.214839 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.214819 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-oauth-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.214839 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.214838 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f20be76-079d-4772-8c17-50a2012848bb-console-serving-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.214969 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.214848 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsr4k\" (UniqueName: \"kubernetes.io/projected/4f20be76-079d-4772-8c17-50a2012848bb-kube-api-access-gsr4k\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.214969 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.214856 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f20be76-079d-4772-8c17-50a2012848bb-console-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 18:59:32.827668 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.827639 2580 generic.go:356] "Generic (PLEG): container finished" podID="4f20be76-079d-4772-8c17-50a2012848bb" containerID="5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a" exitCode=0 Sep 16 18:59:32.828109 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.827684 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd87c846-78cvj" Sep 16 18:59:32.828109 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.827733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd87c846-78cvj" event={"ID":"4f20be76-079d-4772-8c17-50a2012848bb","Type":"ContainerDied","Data":"5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a"} Sep 16 18:59:32.828109 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.827788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd87c846-78cvj" event={"ID":"4f20be76-079d-4772-8c17-50a2012848bb","Type":"ContainerDied","Data":"400cfc512d13933a2429f78e4b9f435691c58a795d32e822b40c99088685a862"} Sep 16 18:59:32.828109 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.827818 2580 scope.go:117] "RemoveContainer" containerID="5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a" Sep 16 18:59:32.836881 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.836865 2580 scope.go:117] "RemoveContainer" containerID="5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a" Sep 16 18:59:32.837141 ip-10-0-128-157 kubenswrapper[2580]: E0916 18:59:32.837123 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a\": container with ID starting with 5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a not found: ID does not exist" containerID="5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a" Sep 16 18:59:32.837189 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.837150 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a"} err="failed to get container status \"5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a\": rpc error: code = NotFound desc = could not find container \"5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a\": container with ID starting with 5cd149a6b4ef05ff70f2be98858d57bff132b42461286d6a1441e350532e591a not found: ID does not exist" Sep 16 18:59:32.850368 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.850346 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bd87c846-78cvj"] Sep 16 18:59:32.853872 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:32.853850 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55bd87c846-78cvj"] Sep 16 18:59:33.401062 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:33.401035 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f20be76-079d-4772-8c17-50a2012848bb" path="/var/lib/kubelet/pods/4f20be76-079d-4772-8c17-50a2012848bb/volumes" Sep 16 18:59:40.713750 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:40.713714 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 18:59:50.713981 ip-10-0-128-157 kubenswrapper[2580]: I0916 18:59:50.713939 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 19:00:00.714712 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:00.714668 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 19:00:10.714653 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:10.714604 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 19:00:20.715346 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:20.715303 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 19:00:31.594034 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:31.593961 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv"] Sep 16 19:00:31.594616 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:31.594206 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" containerID="cri-o://35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d" gracePeriod=30 Sep 16 19:00:31.594616 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:31.594266 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kube-rbac-proxy" containerID="cri-o://411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0" gracePeriod=30 Sep 16 19:00:32.093017 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:32.092988 2580 generic.go:356] "Generic (PLEG): container finished" podID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerID="411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0" exitCode=2 Sep 16 19:00:32.093169 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:32.093076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerDied","Data":"411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0"} Sep 16 19:00:35.535901 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.535879 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 19:00:35.670499 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670432 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-rbac-proxy-sar-config\") pod \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " Sep 16 19:00:35.670499 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670464 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kserve-provision-location\") pod \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " Sep 16 19:00:35.670499 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670484 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv58b\" (UniqueName: \"kubernetes.io/projected/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-api-access-xv58b\") pod \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " Sep 16 19:00:35.670499 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670500 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-cabundle-cert\") pod \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " Sep 16 19:00:35.670807 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670555 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98926bef-23e1-4c62-ad77-c4edc05bf0bb-proxy-tls\") pod \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\" (UID: \"98926bef-23e1-4c62-ad77-c4edc05bf0bb\") " Sep 16 19:00:35.670863 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670809 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98926bef-23e1-4c62-ad77-c4edc05bf0bb" (UID: "98926bef-23e1-4c62-ad77-c4edc05bf0bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:00:35.670863 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98926bef-23e1-4c62-ad77-c4edc05bf0bb" (UID: "98926bef-23e1-4c62-ad77-c4edc05bf0bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:00:35.670934 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.670887 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98926bef-23e1-4c62-ad77-c4edc05bf0bb" (UID: "98926bef-23e1-4c62-ad77-c4edc05bf0bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:00:35.672586 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.672558 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-api-access-xv58b" (OuterVolumeSpecName: "kube-api-access-xv58b") pod "98926bef-23e1-4c62-ad77-c4edc05bf0bb" (UID: "98926bef-23e1-4c62-ad77-c4edc05bf0bb"). InnerVolumeSpecName "kube-api-access-xv58b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:00:35.672682 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.672670 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98926bef-23e1-4c62-ad77-c4edc05bf0bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98926bef-23e1-4c62-ad77-c4edc05bf0bb" (UID: "98926bef-23e1-4c62-ad77-c4edc05bf0bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:00:35.771223 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.771201 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98926bef-23e1-4c62-ad77-c4edc05bf0bb-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:35.771223 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.771221 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:35.771351 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.771231 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:35.771351 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.771240 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv58b\" (UniqueName: \"kubernetes.io/projected/98926bef-23e1-4c62-ad77-c4edc05bf0bb-kube-api-access-xv58b\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:35.771351 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:35.771248 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98926bef-23e1-4c62-ad77-c4edc05bf0bb-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:36.117756 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.117722 2580 generic.go:356] "Generic (PLEG): container finished" podID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerID="35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d" exitCode=0 Sep 16 19:00:36.117871 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.117785 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" Sep 16 19:00:36.117871 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.117796 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerDied","Data":"35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d"} Sep 16 19:00:36.117952 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.117871 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv" event={"ID":"98926bef-23e1-4c62-ad77-c4edc05bf0bb","Type":"ContainerDied","Data":"2f02a753ce367a62fdd21ccfae8c8f673051b4e611969c02f6b6cc1c82764ce8"} Sep 16 19:00:36.117952 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.117897 2580 scope.go:117] "RemoveContainer" containerID="411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0" Sep 16 19:00:36.126911 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.126896 2580 scope.go:117] "RemoveContainer" containerID="35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d" Sep 16 19:00:36.134445 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.134429 2580 scope.go:117] "RemoveContainer" containerID="5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7" Sep 16 19:00:36.140173 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.140153 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv"] Sep 16 19:00:36.142131 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.142116 2580 scope.go:117] "RemoveContainer" containerID="411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0" Sep 16 19:00:36.142419 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:00:36.142402 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0\": container with ID starting with 411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0 not found: ID does not exist" containerID="411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0" Sep 16 19:00:36.142486 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.142425 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0"} err="failed to get container status \"411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0\": rpc error: code = NotFound desc = could not find container \"411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0\": container with ID starting with 411f92f8462eea0923dca77b548c91e0d90d864244a59d304d39d2e04913f2e0 not found: ID does not exist" Sep 16 19:00:36.142486 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.142439 2580 scope.go:117] "RemoveContainer" containerID="35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d" Sep 16 19:00:36.142705 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:00:36.142686 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d\": container with ID starting with 35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d not found: ID does not exist" containerID="35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d" Sep 16 19:00:36.142773 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.142714 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d"} err="failed to get container status \"35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d\": rpc error: code = NotFound desc = could not find container \"35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d\": container with ID starting with 35ebf110fb6302e7d7299d500a23e7d9ec2043371a1369b20419fbdb06a2756d not found: ID does not exist" Sep 16 19:00:36.142773 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.142735 2580 scope.go:117] "RemoveContainer" containerID="5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7" Sep 16 19:00:36.143022 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:00:36.143001 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7\": container with ID starting with 5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7 not found: ID does not exist" containerID="5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7" Sep 16 19:00:36.143022 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.143027 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7"} err="failed to get container status \"5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7\": rpc error: code = NotFound desc = could not find container \"5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7\": container with ID starting with 5b1b3a4cef2f33dbf55701f19b22abc20e96e51ddbb1f3fef44ab0f00bac87c7 not found: ID does not exist" Sep 16 19:00:36.144520 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.144502 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-qblrv"] Sep 16 19:00:36.853428 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853397 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr"] Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853738 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853749 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853762 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f20be76-079d-4772-8c17-50a2012848bb" containerName="console" Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853767 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f20be76-079d-4772-8c17-50a2012848bb" containerName="console" Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853782 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kube-rbac-proxy" Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853787 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kube-rbac-proxy" Sep 16 19:00:36.853792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853794 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="storage-initializer" Sep 16 19:00:36.854001 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853799 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="storage-initializer" Sep 16 19:00:36.854001 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853854 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kserve-container" Sep 16 19:00:36.854001 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853863 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f20be76-079d-4772-8c17-50a2012848bb" containerName="console" Sep 16 19:00:36.854001 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.853871 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" containerName="kube-rbac-proxy" Sep 16 19:00:36.856848 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.856832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:36.859854 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.859819 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\"" Sep 16 19:00:36.860588 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.860565 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Sep 16 19:00:36.874361 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.874344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr"] Sep 16 19:00:36.957416 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.957367 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp"] Sep 16 19:00:36.960820 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.960803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:36.963223 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.963205 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Sep 16 19:00:36.981277 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.981254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp"] Sep 16 19:00:36.982693 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.982668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:36.982777 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.982713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcftn\" (UniqueName: \"kubernetes.io/projected/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-api-access-rcftn\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:36.982830 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.982787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:36.982830 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.982808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:36.982830 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:36.982828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.083652 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.083786 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.083786 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcftn\" (UniqueName: \"kubernetes.io/projected/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-api-access-rcftn\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.083786 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjkp\" (UniqueName: \"kubernetes.io/projected/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-api-access-9wjkp\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.083786 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.083933 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.083933 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.083933 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.083933 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.084129 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.083965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.084282 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.084254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.084589 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.084566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.084672 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.084594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.086499 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.086471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.092747 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.092726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcftn\" (UniqueName: \"kubernetes.io/projected/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-api-access-rcftn\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.167565 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.167509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:37.184982 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.184957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185087 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.184993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185087 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.185022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185087 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.185052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185087 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.185078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjkp\" (UniqueName: \"kubernetes.io/projected/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-api-access-9wjkp\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185460 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.185421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185671 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.185649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.185766 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.185694 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.188179 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.188148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.194604 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.194568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjkp\" (UniqueName: \"kubernetes.io/projected/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-api-access-9wjkp\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-w48gp\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.270265 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.270235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:37.335248 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.335206 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:00:37.343291 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.343272 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:00:37.400815 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.400793 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98926bef-23e1-4c62-ad77-c4edc05bf0bb" path="/var/lib/kubelet/pods/98926bef-23e1-4c62-ad77-c4edc05bf0bb/volumes" Sep 16 19:00:37.449885 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.449854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr"] Sep 16 19:00:37.452076 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:00:37.452053 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cf9588_4f2f_4d89_9cd5_c6b85fe2c4d1.slice/crio-21ed475a74f71153c0c24154d07f75e2b759f8d8c8b2862006f66c18d8ffd5b8 WatchSource:0}: Error finding container 21ed475a74f71153c0c24154d07f75e2b759f8d8c8b2862006f66c18d8ffd5b8: Status 404 returned error can't find the container with id 21ed475a74f71153c0c24154d07f75e2b759f8d8c8b2862006f66c18d8ffd5b8 Sep 16 19:00:37.553436 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:37.553411 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp"] Sep 16 19:00:37.555003 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:00:37.554981 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2fb136_f4f2_4cb0_953e_acf30204c9a7.slice/crio-d5503b9dabb558934b6890196b8d3ceaf486909a5b9695d233eda12154b8caf2 WatchSource:0}: Error finding container d5503b9dabb558934b6890196b8d3ceaf486909a5b9695d233eda12154b8caf2: Status 404 returned error can't find the container with id d5503b9dabb558934b6890196b8d3ceaf486909a5b9695d233eda12154b8caf2 Sep 16 19:00:38.139550 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:38.139507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerStarted","Data":"8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42"} Sep 16 19:00:38.140030 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:38.139611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerStarted","Data":"d5503b9dabb558934b6890196b8d3ceaf486909a5b9695d233eda12154b8caf2"} Sep 16 19:00:38.141251 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:38.141222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerStarted","Data":"a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d"} Sep 16 19:00:38.141405 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:38.141279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerStarted","Data":"21ed475a74f71153c0c24154d07f75e2b759f8d8c8b2862006f66c18d8ffd5b8"} Sep 16 19:00:42.160453 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:42.160422 2580 generic.go:356] "Generic (PLEG): container finished" podID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerID="8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42" exitCode=0 Sep 16 19:00:42.160869 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:42.160499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerDied","Data":"8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42"} Sep 16 19:00:42.162081 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:42.162058 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerID="a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d" exitCode=0 Sep 16 19:00:42.162186 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:42.162091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerDied","Data":"a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d"} Sep 16 19:00:43.169519 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.169480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerStarted","Data":"ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1"} Sep 16 19:00:43.169944 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.169552 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerStarted","Data":"66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5"} Sep 16 19:00:43.169944 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.169924 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:43.170052 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.170036 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:43.171197 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.171171 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:00:43.171764 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.171748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerStarted","Data":"b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9"} Sep 16 19:00:43.171838 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.171791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerStarted","Data":"11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f"} Sep 16 19:00:43.172033 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.172018 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:43.172091 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.172073 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:43.173048 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.173022 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:00:43.195122 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.195083 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podStartSLOduration=3.079256225 podStartE2EDuration="7.195071935s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.11581571s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:43.193869072 +0000 UTC m=+906.316716579" watchObservedRunningTime="2026-09-16 19:00:43.195071935 +0000 UTC m=+906.317919478" Sep 16 19:00:43.247009 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:43.246956 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podStartSLOduration=3.271405576 podStartE2EDuration="7.246938331s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975532755s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:43.244425379 +0000 UTC m=+906.367272885" watchObservedRunningTime="2026-09-16 19:00:43.246938331 +0000 UTC m=+906.369785838" Sep 16 19:00:44.176563 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:44.176511 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:00:44.176982 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:44.176745 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:00:46.916102 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:46.916068 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq"] Sep 16 19:00:46.916580 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:46.916300 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" containerID="cri-o://bf66445d34149264f058a492ea7188aaf45848a2291435dbf8d9e4aa1c50e3c0" gracePeriod=30 Sep 16 19:00:46.916580 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:46.916332 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kube-rbac-proxy" containerID="cri-o://e1b1992277623da9d656b713428002cb30d8066f27ca72bd56cf7db0b9dec0fd" gracePeriod=30 Sep 16 19:00:47.192628 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:47.192536 2580 generic.go:356] "Generic (PLEG): container finished" podID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerID="e1b1992277623da9d656b713428002cb30d8066f27ca72bd56cf7db0b9dec0fd" exitCode=2 Sep 16 19:00:47.192628 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:47.192586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerDied","Data":"e1b1992277623da9d656b713428002cb30d8066f27ca72bd56cf7db0b9dec0fd"} Sep 16 19:00:49.181415 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:49.181372 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:00:49.181859 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:49.181569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:00:49.181955 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:49.181928 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:00:49.181997 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:49.181981 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:00:50.709493 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:50.709451 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Sep 16 19:00:50.713907 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:50.713885 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 16 19:00:51.212193 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.212158 2580 generic.go:356] "Generic (PLEG): container finished" podID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerID="bf66445d34149264f058a492ea7188aaf45848a2291435dbf8d9e4aa1c50e3c0" exitCode=0 Sep 16 19:00:51.212331 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.212217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerDied","Data":"bf66445d34149264f058a492ea7188aaf45848a2291435dbf8d9e4aa1c50e3c0"} Sep 16 19:00:51.255676 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.255652 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 19:00:51.299712 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.299690 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls\") pod \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " Sep 16 19:00:51.299833 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.299748 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kserve-provision-location\") pod \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " Sep 16 19:00:51.299833 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.299775 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-rbac-proxy-sar-config\") pod \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " Sep 16 19:00:51.299961 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.299853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfl68\" (UniqueName: \"kubernetes.io/projected/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-api-access-jfl68\") pod \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " Sep 16 19:00:51.299961 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.299911 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-cabundle-cert\") pod \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\" (UID: \"3624fc6d-dc56-4b96-b4b2-0109e7940aa5\") " Sep 16 19:00:51.300094 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.300065 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3624fc6d-dc56-4b96-b4b2-0109e7940aa5" (UID: "3624fc6d-dc56-4b96-b4b2-0109e7940aa5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:00:51.300169 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.300154 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:51.300216 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.300173 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3624fc6d-dc56-4b96-b4b2-0109e7940aa5" (UID: "3624fc6d-dc56-4b96-b4b2-0109e7940aa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:00:51.300301 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.300280 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3624fc6d-dc56-4b96-b4b2-0109e7940aa5" (UID: "3624fc6d-dc56-4b96-b4b2-0109e7940aa5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:00:51.302141 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.302122 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-api-access-jfl68" (OuterVolumeSpecName: "kube-api-access-jfl68") pod "3624fc6d-dc56-4b96-b4b2-0109e7940aa5" (UID: "3624fc6d-dc56-4b96-b4b2-0109e7940aa5"). InnerVolumeSpecName "kube-api-access-jfl68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:00:51.302350 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.302335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3624fc6d-dc56-4b96-b4b2-0109e7940aa5" (UID: "3624fc6d-dc56-4b96-b4b2-0109e7940aa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:00:51.401018 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.400990 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:51.401018 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.401018 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfl68\" (UniqueName: \"kubernetes.io/projected/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-kube-api-access-jfl68\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:51.401179 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.401033 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:51.401179 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:51.401046 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3624fc6d-dc56-4b96-b4b2-0109e7940aa5-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:00:52.220356 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.220322 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" Sep 16 19:00:52.220775 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.220331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq" event={"ID":"3624fc6d-dc56-4b96-b4b2-0109e7940aa5","Type":"ContainerDied","Data":"47377aabf2dce18f35eade2f8b1d6a3d9c878edb81468d0704fb4a6304243f4d"} Sep 16 19:00:52.220775 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.220479 2580 scope.go:117] "RemoveContainer" containerID="e1b1992277623da9d656b713428002cb30d8066f27ca72bd56cf7db0b9dec0fd" Sep 16 19:00:52.229182 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.229162 2580 scope.go:117] "RemoveContainer" containerID="bf66445d34149264f058a492ea7188aaf45848a2291435dbf8d9e4aa1c50e3c0" Sep 16 19:00:52.236997 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.236980 2580 scope.go:117] "RemoveContainer" containerID="bf677e4710a19e377a1e4556a87ef96738cccb02b964de5984fc583f731a14a2" Sep 16 19:00:52.240085 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.240064 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq"] Sep 16 19:00:52.244196 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:52.244175 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-1ba12-predictor-6bb777b994-rsmbq"] Sep 16 19:00:53.403448 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:53.403417 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" path="/var/lib/kubelet/pods/3624fc6d-dc56-4b96-b4b2-0109e7940aa5/volumes" Sep 16 19:00:59.182439 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:59.182399 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:00:59.182875 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:00:59.182440 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:01:09.182890 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:09.182845 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:01:09.183354 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:09.182848 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:01:19.182360 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:19.182322 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:01:19.182360 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:19.182343 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:01:29.182579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:29.182536 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:01:29.182968 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:29.182536 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:01:39.182639 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:39.182600 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:01:39.183000 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:39.182612 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:01:49.182589 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:49.182546 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:01:49.184862 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:49.182546 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:01:50.396335 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:50.396285 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:01:59.182996 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:01:59.182920 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:02:00.397048 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:00.397017 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:02:06.774952 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:06.774917 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr"] Sep 16 19:02:06.775361 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:06.775144 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" containerID="cri-o://11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f" gracePeriod=30 Sep 16 19:02:06.775361 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:06.775191 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kube-rbac-proxy" containerID="cri-o://b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9" gracePeriod=30 Sep 16 19:02:06.821550 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:06.821520 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp"] Sep 16 19:02:06.822090 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:06.822058 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" containerID="cri-o://66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5" gracePeriod=30 Sep 16 19:02:06.822572 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:06.822523 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kube-rbac-proxy" containerID="cri-o://ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1" gracePeriod=30 Sep 16 19:02:07.554243 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:07.554205 2580 generic.go:356] "Generic (PLEG): container finished" podID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerID="ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1" exitCode=2 Sep 16 19:02:07.554443 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:07.554281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerDied","Data":"ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1"} Sep 16 19:02:07.556398 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:07.556355 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerID="b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9" exitCode=2 Sep 16 19:02:07.556520 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:07.556403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerDied","Data":"b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9"} Sep 16 19:02:09.177447 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:09.177404 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused" Sep 16 19:02:09.177840 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:09.177403 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused" Sep 16 19:02:09.182433 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:09.182409 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 16 19:02:10.396559 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:10.396519 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 19:02:10.997343 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:10.997316 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:02:11.099990 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.099922 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wjkp\" (UniqueName: \"kubernetes.io/projected/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-api-access-9wjkp\") pod \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " Sep 16 19:02:11.099990 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.099961 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kserve-provision-location\") pod \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " Sep 16 19:02:11.100237 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.100001 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-rbac-proxy-sar-config\") pod \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " Sep 16 19:02:11.100237 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.100017 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-proxy-tls\") pod \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " Sep 16 19:02:11.100237 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.100069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-cabundle-cert\") pod \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\" (UID: \"ca2fb136-f4f2-4cb0-953e-acf30204c9a7\") " Sep 16 19:02:11.100422 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.100339 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca2fb136-f4f2-4cb0-953e-acf30204c9a7" (UID: "ca2fb136-f4f2-4cb0-953e-acf30204c9a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:02:11.100485 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.100423 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca2fb136-f4f2-4cb0-953e-acf30204c9a7" (UID: "ca2fb136-f4f2-4cb0-953e-acf30204c9a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:02:11.100704 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.100682 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca2fb136-f4f2-4cb0-953e-acf30204c9a7" (UID: "ca2fb136-f4f2-4cb0-953e-acf30204c9a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:02:11.102237 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.102216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca2fb136-f4f2-4cb0-953e-acf30204c9a7" (UID: "ca2fb136-f4f2-4cb0-953e-acf30204c9a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:02:11.102314 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.102263 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-api-access-9wjkp" (OuterVolumeSpecName: "kube-api-access-9wjkp") pod "ca2fb136-f4f2-4cb0-953e-acf30204c9a7" (UID: "ca2fb136-f4f2-4cb0-953e-acf30204c9a7"). InnerVolumeSpecName "kube-api-access-9wjkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:02:11.122801 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.122779 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:02:11.200833 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.200807 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-proxy-tls\") pod \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " Sep 16 19:02:11.200943 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.200851 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-rbac-proxy-sar-config\") pod \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " Sep 16 19:02:11.200943 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.200898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kserve-provision-location\") pod \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " Sep 16 19:02:11.200943 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.200932 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcftn\" (UniqueName: \"kubernetes.io/projected/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-api-access-rcftn\") pod \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " Sep 16 19:02:11.201099 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.200950 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-cabundle-cert\") pod \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\" (UID: \"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1\") " Sep 16 19:02:11.201155 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201121 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjkp\" (UniqueName: \"kubernetes.io/projected/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-api-access-9wjkp\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.201155 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201139 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.201258 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201155 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.201258 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201172 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.201258 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201185 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca2fb136-f4f2-4cb0-953e-acf30204c9a7-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.201258 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" (UID: "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:02:11.201415 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201293 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" (UID: "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:02:11.201415 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.201317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" (UID: "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:02:11.203038 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.203018 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-api-access-rcftn" (OuterVolumeSpecName: "kube-api-access-rcftn") pod "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" (UID: "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1"). InnerVolumeSpecName "kube-api-access-rcftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:02:11.203038 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.203029 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" (UID: "f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:02:11.302105 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.302073 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.302105 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.302103 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.302105 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.302111 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.302267 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.302120 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcftn\" (UniqueName: \"kubernetes.io/projected/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-kube-api-access-rcftn\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.302267 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.302129 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:02:11.577121 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.577096 2580 generic.go:356] "Generic (PLEG): container finished" podID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerID="66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5" exitCode=0 Sep 16 19:02:11.577527 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.577174 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" Sep 16 19:02:11.577527 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.577186 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerDied","Data":"66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5"} Sep 16 19:02:11.577527 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.577255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp" event={"ID":"ca2fb136-f4f2-4cb0-953e-acf30204c9a7","Type":"ContainerDied","Data":"d5503b9dabb558934b6890196b8d3ceaf486909a5b9695d233eda12154b8caf2"} Sep 16 19:02:11.577527 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.577293 2580 scope.go:117] "RemoveContainer" containerID="ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1" Sep 16 19:02:11.579481 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.579336 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerID="11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f" exitCode=0 Sep 16 19:02:11.579481 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.579373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerDied","Data":"11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f"} Sep 16 19:02:11.579481 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.579419 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" Sep 16 19:02:11.579481 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.579454 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr" event={"ID":"f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1","Type":"ContainerDied","Data":"21ed475a74f71153c0c24154d07f75e2b759f8d8c8b2862006f66c18d8ffd5b8"} Sep 16 19:02:11.585909 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.585893 2580 scope.go:117] "RemoveContainer" containerID="66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5" Sep 16 19:02:11.593351 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.593306 2580 scope.go:117] "RemoveContainer" containerID="8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42" Sep 16 19:02:11.595010 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.594992 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp"] Sep 16 19:02:11.599620 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.599600 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-w48gp"] Sep 16 19:02:11.601958 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.601944 2580 scope.go:117] "RemoveContainer" containerID="ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1" Sep 16 19:02:11.602199 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:11.602182 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1\": container with ID starting with ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1 not found: ID does not exist" containerID="ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1" Sep 16 19:02:11.602246 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.602206 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1"} err="failed to get container status \"ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1\": rpc error: code = NotFound desc = could not find container \"ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1\": container with ID starting with ac5454f1434574f36227ca6f37f3b27ff28898d9fe9cbd40f97b8deca50cabc1 not found: ID does not exist" Sep 16 19:02:11.602246 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.602221 2580 scope.go:117] "RemoveContainer" containerID="66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5" Sep 16 19:02:11.602546 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:11.602526 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5\": container with ID starting with 66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5 not found: ID does not exist" containerID="66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5" Sep 16 19:02:11.602632 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.602565 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5"} err="failed to get container status \"66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5\": rpc error: code = NotFound desc = could not find container \"66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5\": container with ID starting with 66ee84f8d49d45fbd1e721397f669df1b300aef98d8d0e0d2df4c5daf85cd0e5 not found: ID does not exist" Sep 16 19:02:11.602632 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.602589 2580 scope.go:117] "RemoveContainer" containerID="8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42" Sep 16 19:02:11.602854 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:11.602835 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42\": container with ID starting with 8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42 not found: ID does not exist" containerID="8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42" Sep 16 19:02:11.602901 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.602861 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42"} err="failed to get container status \"8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42\": rpc error: code = NotFound desc = could not find container \"8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42\": container with ID starting with 8c1e4737b9a57d6708f06db4649d94ca7dd794b02136382150d818be36b3ab42 not found: ID does not exist" Sep 16 19:02:11.602901 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.602874 2580 scope.go:117] "RemoveContainer" containerID="b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9" Sep 16 19:02:11.610070 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.610054 2580 scope.go:117] "RemoveContainer" containerID="11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f" Sep 16 19:02:11.617645 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.617627 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr"] Sep 16 19:02:11.617895 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.617879 2580 scope.go:117] "RemoveContainer" containerID="a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d" Sep 16 19:02:11.625750 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.625469 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-8hjwr"] Sep 16 19:02:11.632363 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.632347 2580 scope.go:117] "RemoveContainer" containerID="b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9" Sep 16 19:02:11.632728 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:11.632687 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9\": container with ID starting with b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9 not found: ID does not exist" containerID="b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9" Sep 16 19:02:11.632787 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.632749 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9"} err="failed to get container status \"b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9\": rpc error: code = NotFound desc = could not find container \"b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9\": container with ID starting with b616512527c941f4bd1e8c29b01f4f5f8ab310ace22e3fda62c72c993f8265f9 not found: ID does not exist" Sep 16 19:02:11.632830 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.632790 2580 scope.go:117] "RemoveContainer" containerID="11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f" Sep 16 19:02:11.633156 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:11.633131 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f\": container with ID starting with 11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f not found: ID does not exist" containerID="11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f" Sep 16 19:02:11.633219 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.633165 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f"} err="failed to get container status \"11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f\": rpc error: code = NotFound desc = could not find container \"11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f\": container with ID starting with 11e3b207eb7340894da54f430c96b38a9dfe76c2e81c69f566fca45bf3287f8f not found: ID does not exist" Sep 16 19:02:11.633219 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.633192 2580 scope.go:117] "RemoveContainer" containerID="a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d" Sep 16 19:02:11.633497 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:11.633469 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d\": container with ID starting with a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d not found: ID does not exist" containerID="a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d" Sep 16 19:02:11.633581 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:11.633509 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d"} err="failed to get container status \"a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d\": rpc error: code = NotFound desc = could not find container \"a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d\": container with ID starting with a4b10eac3147141fed0809f1d045a1bb7472f37c92c8d2709b1e83cbfe98374d not found: ID does not exist" Sep 16 19:02:13.402917 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:13.402883 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" path="/var/lib/kubelet/pods/ca2fb136-f4f2-4cb0-953e-acf30204c9a7/volumes" Sep 16 19:02:13.403348 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:13.403335 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" path="/var/lib/kubelet/pods/f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1/volumes" Sep 16 19:02:17.133554 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.133519 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b"] Sep 16 19:02:17.134002 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.133988 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="storage-initializer" Sep 16 19:02:17.134077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134006 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="storage-initializer" Sep 16 19:02:17.134077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134020 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" Sep 16 19:02:17.134077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134028 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" Sep 16 19:02:17.134077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134040 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kube-rbac-proxy" Sep 16 19:02:17.134077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134048 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kube-rbac-proxy" Sep 16 19:02:17.134077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134072 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kube-rbac-proxy" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134081 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kube-rbac-proxy" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134092 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="storage-initializer" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134100 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="storage-initializer" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134115 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134123 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134136 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134144 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134161 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="storage-initializer" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134170 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="storage-initializer" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134179 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kube-rbac-proxy" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134187 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kube-rbac-proxy" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134263 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kube-rbac-proxy" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134274 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134285 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134297 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2fb136-f4f2-4cb0-953e-acf30204c9a7" containerName="kserve-container" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134307 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cf9588-4f2f-4d89-9cd5-c6b85fe2c4d1" containerName="kube-rbac-proxy" Sep 16 19:02:17.134370 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.134315 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3624fc6d-dc56-4b96-b4b2-0109e7940aa5" containerName="kube-rbac-proxy" Sep 16 19:02:17.139699 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.139678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.143604 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.143580 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:02:17.143714 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.143609 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-8f4cb6-predictor-serving-cert\"" Sep 16 19:02:17.143714 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.143624 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:02:17.144179 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.144157 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:02:17.144277 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.144260 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\"" Sep 16 19:02:17.144340 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.144282 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:02:17.144340 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.144305 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-8f4cb6-kube-rbac-proxy-sar-config\"" Sep 16 19:02:17.154492 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.154472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b"] Sep 16 19:02:17.243416 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.243367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrnkn\" (UniqueName: \"kubernetes.io/projected/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-api-access-vrnkn\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.243544 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.243429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-cabundle-cert\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.243544 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.243460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kserve-provision-location\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.243544 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.243502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-rbac-proxy-sar-config\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.243544 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.243526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-proxy-tls\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.344440 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.344414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kserve-provision-location\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.344532 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.344463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-rbac-proxy-sar-config\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.344532 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.344483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-proxy-tls\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.344532 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.344510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrnkn\" (UniqueName: \"kubernetes.io/projected/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-api-access-vrnkn\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.344532 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.344532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-cabundle-cert\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.344853 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.344827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kserve-provision-location\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.345132 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.345113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-cabundle-cert\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.345224 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.345115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-rbac-proxy-sar-config\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.346979 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.346955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-proxy-tls\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.354906 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.354887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrnkn\" (UniqueName: \"kubernetes.io/projected/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-api-access-vrnkn\") pod \"isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.450593 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.450538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:17.727277 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:17.727215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b"] Sep 16 19:02:17.730574 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:02:17.730545 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791e0be5_0d4f_45d3_8e1d_86c1d89bc978.slice/crio-1b184c309f3e629c35bfb79e1b2d71d3b1772bc785d2824a257d8e22283258c9 WatchSource:0}: Error finding container 1b184c309f3e629c35bfb79e1b2d71d3b1772bc785d2824a257d8e22283258c9: Status 404 returned error can't find the container with id 1b184c309f3e629c35bfb79e1b2d71d3b1772bc785d2824a257d8e22283258c9 Sep 16 19:02:18.613124 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:18.613087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerStarted","Data":"bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01"} Sep 16 19:02:18.613578 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:18.613150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerStarted","Data":"1b184c309f3e629c35bfb79e1b2d71d3b1772bc785d2824a257d8e22283258c9"} Sep 16 19:02:22.630475 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:22.630442 2580 generic.go:356] "Generic (PLEG): container finished" podID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerID="bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01" exitCode=0 Sep 16 19:02:22.630899 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:22.630518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerDied","Data":"bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01"} Sep 16 19:02:23.637155 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:23.637121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerStarted","Data":"58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6"} Sep 16 19:02:23.637578 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:23.637178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerStarted","Data":"969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7"} Sep 16 19:02:23.637578 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:23.637510 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:23.637673 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:23.637647 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:23.639015 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:23.638989 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:02:23.676640 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:23.676599 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podStartSLOduration=2.609244977 podStartE2EDuration="6.676584939s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.067339962s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:23.674117415 +0000 UTC m=+1006.796964923" watchObservedRunningTime="2026-09-16 19:02:23.676584939 +0000 UTC m=+1006.799432445" Sep 16 19:02:24.641150 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:24.641108 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:02:25.645470 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:25.645431 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:02:30.650179 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:30.650153 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:02:30.650596 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:30.650560 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:02:40.651594 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:40.651558 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:02:47.460190 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.460163 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx"] Sep 16 19:02:47.464718 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.464698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.468169 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.468151 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-87fdd-predictor-serving-cert\"" Sep 16 19:02:47.468931 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.468913 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-87fdd-kube-rbac-proxy-sar-config\"" Sep 16 19:02:47.480165 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.480140 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx"] Sep 16 19:02:47.571133 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.571099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.571133 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.571144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqgc\" (UniqueName: \"kubernetes.io/projected/82742154-de84-48be-99fc-59f5a30857d3-kube-api-access-dhqgc\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.571355 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.571179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82742154-de84-48be-99fc-59f5a30857d3-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.671664 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.671636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.671804 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.671670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqgc\" (UniqueName: \"kubernetes.io/projected/82742154-de84-48be-99fc-59f5a30857d3-kube-api-access-dhqgc\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.671804 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.671693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82742154-de84-48be-99fc-59f5a30857d3-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.671905 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:47.671796 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-serving-cert: secret "message-dumper-raw-87fdd-predictor-serving-cert" not found Sep 16 19:02:47.671905 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:02:47.671864 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls podName:82742154-de84-48be-99fc-59f5a30857d3 nodeName:}" failed. No retries permitted until 2026-09-16 19:02:48.171843817 +0000 UTC m=+1031.294691316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls") pod "message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" (UID: "82742154-de84-48be-99fc-59f5a30857d3") : secret "message-dumper-raw-87fdd-predictor-serving-cert" not found Sep 16 19:02:47.672250 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.672233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82742154-de84-48be-99fc-59f5a30857d3-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:47.682109 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:47.682084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqgc\" (UniqueName: \"kubernetes.io/projected/82742154-de84-48be-99fc-59f5a30857d3-kube-api-access-dhqgc\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:48.175505 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:48.175474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:48.178039 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:48.178018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls\") pod \"message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:48.380461 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:48.380422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:48.654111 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:48.654083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx"] Sep 16 19:02:48.655978 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:02:48.655951 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82742154_de84_48be_99fc_59f5a30857d3.slice/crio-629ebb28b44a8cf0502337f1246674ebf18f2dbe0db9a4b126d0ffa682ddd193 WatchSource:0}: Error finding container 629ebb28b44a8cf0502337f1246674ebf18f2dbe0db9a4b126d0ffa682ddd193: Status 404 returned error can't find the container with id 629ebb28b44a8cf0502337f1246674ebf18f2dbe0db9a4b126d0ffa682ddd193 Sep 16 19:02:48.658211 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:48.658194 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:02:48.743061 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:48.743004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" event={"ID":"82742154-de84-48be-99fc-59f5a30857d3","Type":"ContainerStarted","Data":"629ebb28b44a8cf0502337f1246674ebf18f2dbe0db9a4b126d0ffa682ddd193"} Sep 16 19:02:50.651003 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:50.650963 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:02:50.753858 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:50.753827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" event={"ID":"82742154-de84-48be-99fc-59f5a30857d3","Type":"ContainerStarted","Data":"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c"} Sep 16 19:02:50.754002 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:50.753882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" event={"ID":"82742154-de84-48be-99fc-59f5a30857d3","Type":"ContainerStarted","Data":"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32"} Sep 16 19:02:50.754055 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:50.753994 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:50.776189 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:50.776147 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" podStartSLOduration=2.700224022 podStartE2EDuration="3.776134264s" totalImagesPullingTime="1.075910242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:50.774577929 +0000 UTC m=+1033.897425440" watchObservedRunningTime="2026-09-16 19:02:50.776134264 +0000 UTC m=+1033.898981770" Sep 16 19:02:51.758301 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:51.758269 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:51.759870 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:51.759848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:02:58.771594 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:02:58.771567 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:03:00.650920 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:03:00.650882 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:03:10.651457 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:03:10.651419 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:03:20.650913 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:03:20.650874 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:03:30.651650 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:03:30.651573 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:03:40.651524 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:03:40.651497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:04:18.396809 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:18.396780 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b"] Sep 16 19:04:18.397217 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:18.396998 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" containerID="cri-o://969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7" gracePeriod=30 Sep 16 19:04:18.397217 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:18.397032 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kube-rbac-proxy" containerID="cri-o://58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6" gracePeriod=30 Sep 16 19:04:19.160369 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:19.160332 2580 generic.go:356] "Generic (PLEG): container finished" podID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerID="58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6" exitCode=2 Sep 16 19:04:19.160564 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:19.160407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerDied","Data":"58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6"} Sep 16 19:04:20.646086 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:20.646042 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused" Sep 16 19:04:20.651397 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:20.651349 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 19:04:22.140850 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.140828 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:04:22.175155 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.175092 2580 generic.go:356] "Generic (PLEG): container finished" podID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerID="969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7" exitCode=0 Sep 16 19:04:22.175281 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.175161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerDied","Data":"969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7"} Sep 16 19:04:22.175281 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.175199 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" Sep 16 19:04:22.175281 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.175211 2580 scope.go:117] "RemoveContainer" containerID="58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6" Sep 16 19:04:22.175281 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.175202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b" event={"ID":"791e0be5-0d4f-45d3-8e1d-86c1d89bc978","Type":"ContainerDied","Data":"1b184c309f3e629c35bfb79e1b2d71d3b1772bc785d2824a257d8e22283258c9"} Sep 16 19:04:22.183441 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.183425 2580 scope.go:117] "RemoveContainer" containerID="969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7" Sep 16 19:04:22.190608 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.190589 2580 scope.go:117] "RemoveContainer" containerID="bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01" Sep 16 19:04:22.197929 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.197913 2580 scope.go:117] "RemoveContainer" containerID="58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6" Sep 16 19:04:22.198192 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:22.198171 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6\": container with ID starting with 58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6 not found: ID does not exist" containerID="58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6" Sep 16 19:04:22.198235 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.198202 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6"} err="failed to get container status \"58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6\": rpc error: code = NotFound desc = could not find container \"58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6\": container with ID starting with 58e1a901a99694558a9960a5b745de804e5906ca529b34aa62626a644deffff6 not found: ID does not exist" Sep 16 19:04:22.198235 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.198219 2580 scope.go:117] "RemoveContainer" containerID="969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7" Sep 16 19:04:22.198512 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:22.198495 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7\": container with ID starting with 969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7 not found: ID does not exist" containerID="969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7" Sep 16 19:04:22.198579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.198518 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7"} err="failed to get container status \"969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7\": rpc error: code = NotFound desc = could not find container \"969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7\": container with ID starting with 969d6ab8f708a0465598ab63affdb87b6d3dfbd60385072c06fcd58d5c942ba7 not found: ID does not exist" Sep 16 19:04:22.198579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.198534 2580 scope.go:117] "RemoveContainer" containerID="bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01" Sep 16 19:04:22.198775 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:22.198757 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01\": container with ID starting with bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01 not found: ID does not exist" containerID="bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01" Sep 16 19:04:22.198822 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.198779 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01"} err="failed to get container status \"bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01\": rpc error: code = NotFound desc = could not find container \"bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01\": container with ID starting with bc85d90c33d35acdfcabe9319eb21084c2252e76fddfb028769661d811e05c01 not found: ID does not exist" Sep 16 19:04:22.254331 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254310 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-rbac-proxy-sar-config\") pod \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " Sep 16 19:04:22.254433 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kserve-provision-location\") pod \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " Sep 16 19:04:22.254433 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254408 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrnkn\" (UniqueName: \"kubernetes.io/projected/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-api-access-vrnkn\") pod \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " Sep 16 19:04:22.254433 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254426 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-proxy-tls\") pod \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " Sep 16 19:04:22.254586 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254448 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-cabundle-cert\") pod \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\" (UID: \"791e0be5-0d4f-45d3-8e1d-86c1d89bc978\") " Sep 16 19:04:22.254786 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254761 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "791e0be5-0d4f-45d3-8e1d-86c1d89bc978" (UID: "791e0be5-0d4f-45d3-8e1d-86c1d89bc978"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:04:22.254870 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "791e0be5-0d4f-45d3-8e1d-86c1d89bc978" (UID: "791e0be5-0d4f-45d3-8e1d-86c1d89bc978"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:22.254870 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.254847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "791e0be5-0d4f-45d3-8e1d-86c1d89bc978" (UID: "791e0be5-0d4f-45d3-8e1d-86c1d89bc978"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:22.256432 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.256409 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-api-access-vrnkn" (OuterVolumeSpecName: "kube-api-access-vrnkn") pod "791e0be5-0d4f-45d3-8e1d-86c1d89bc978" (UID: "791e0be5-0d4f-45d3-8e1d-86c1d89bc978"). InnerVolumeSpecName "kube-api-access-vrnkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:04:22.256540 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.256520 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "791e0be5-0d4f-45d3-8e1d-86c1d89bc978" (UID: "791e0be5-0d4f-45d3-8e1d-86c1d89bc978"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:04:22.355610 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.355587 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:22.355610 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.355607 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:22.355729 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.355615 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrnkn\" (UniqueName: \"kubernetes.io/projected/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-kube-api-access-vrnkn\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:22.355729 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.355625 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:22.355729 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.355635 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791e0be5-0d4f-45d3-8e1d-86c1d89bc978-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:22.498264 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.498240 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b"] Sep 16 19:04:22.504503 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:22.504484 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8f4cb6-predictor-7fb56ccb67-7mn2b"] Sep 16 19:04:23.400930 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.400904 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" path="/var/lib/kubelet/pods/791e0be5-0d4f-45d3-8e1d-86c1d89bc978/volumes" Sep 16 19:04:23.546088 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546055 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62"] Sep 16 19:04:23.546426 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546412 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" Sep 16 19:04:23.546476 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546428 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" Sep 16 19:04:23.546476 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546437 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kube-rbac-proxy" Sep 16 19:04:23.546476 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546443 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kube-rbac-proxy" Sep 16 19:04:23.546476 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546450 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="storage-initializer" Sep 16 19:04:23.546476 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546455 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="storage-initializer" Sep 16 19:04:23.546626 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546517 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kube-rbac-proxy" Sep 16 19:04:23.546626 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.546529 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="791e0be5-0d4f-45d3-8e1d-86c1d89bc978" containerName="kserve-container" Sep 16 19:04:23.551066 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.551037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.553418 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.553374 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a6bd8f-predictor-serving-cert\"" Sep 16 19:04:23.553537 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.553423 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-a6bd8f\"" Sep 16 19:04:23.553537 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.553431 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-a6bd8f-dockercfg-wc4c8\"" Sep 16 19:04:23.553537 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.553487 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:04:23.553537 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.553424 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a6bd8f-kube-rbac-proxy-sar-config\"" Sep 16 19:04:23.560317 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.560293 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62"] Sep 16 19:04:23.664577 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.664513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mww\" (UniqueName: \"kubernetes.io/projected/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-api-access-t6mww\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.664577 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.664552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5da5146-5864-4e91-b48e-57f372ad3bd8-kserve-provision-location\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.664577 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.664569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-cabundle-cert\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.664751 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.664627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.664751 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.664715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.765726 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.765694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mww\" (UniqueName: \"kubernetes.io/projected/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-api-access-t6mww\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.765850 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.765737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5da5146-5864-4e91-b48e-57f372ad3bd8-kserve-provision-location\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.765850 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.765754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-cabundle-cert\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.765850 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.765775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.765850 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.765843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.766053 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:23.765963 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-serving-cert: secret "isvc-init-fail-a6bd8f-predictor-serving-cert" not found Sep 16 19:04:23.766053 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:23.766040 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls podName:b5da5146-5864-4e91-b48e-57f372ad3bd8 nodeName:}" failed. No retries permitted until 2026-09-16 19:04:24.266019656 +0000 UTC m=+1127.388867156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls") pod "isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" (UID: "b5da5146-5864-4e91-b48e-57f372ad3bd8") : secret "isvc-init-fail-a6bd8f-predictor-serving-cert" not found Sep 16 19:04:23.766174 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.766057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5da5146-5864-4e91-b48e-57f372ad3bd8-kserve-provision-location\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.766513 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.766490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-cabundle-cert\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.766567 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.766551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:23.774291 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:23.774265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mww\" (UniqueName: \"kubernetes.io/projected/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-api-access-t6mww\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:24.269907 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:24.269875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:24.272330 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:24.272304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls\") pod \"isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:24.462221 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:24.462190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:24.737550 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:24.737528 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62"] Sep 16 19:04:24.739801 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:04:24.739778 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da5146_5864_4e91_b48e_57f372ad3bd8.slice/crio-775af00b209ff9a950f2fd3c2828c1274b6b74166ed0e236862f15be163242f4 WatchSource:0}: Error finding container 775af00b209ff9a950f2fd3c2828c1274b6b74166ed0e236862f15be163242f4: Status 404 returned error can't find the container with id 775af00b209ff9a950f2fd3c2828c1274b6b74166ed0e236862f15be163242f4 Sep 16 19:04:25.193986 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:25.193949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" event={"ID":"b5da5146-5864-4e91-b48e-57f372ad3bd8","Type":"ContainerStarted","Data":"07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d"} Sep 16 19:04:25.194136 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:25.194019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" event={"ID":"b5da5146-5864-4e91-b48e-57f372ad3bd8","Type":"ContainerStarted","Data":"775af00b209ff9a950f2fd3c2828c1274b6b74166ed0e236862f15be163242f4"} Sep 16 19:04:30.218136 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:30.218108 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62_b5da5146-5864-4e91-b48e-57f372ad3bd8/storage-initializer/0.log" Sep 16 19:04:30.218536 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:30.218144 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerID="07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d" exitCode=1 Sep 16 19:04:30.218536 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:30.218223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" event={"ID":"b5da5146-5864-4e91-b48e-57f372ad3bd8","Type":"ContainerDied","Data":"07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d"} Sep 16 19:04:31.224144 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:31.224118 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62_b5da5146-5864-4e91-b48e-57f372ad3bd8/storage-initializer/0.log" Sep 16 19:04:31.224521 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:31.224195 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" event={"ID":"b5da5146-5864-4e91-b48e-57f372ad3bd8","Type":"ContainerStarted","Data":"5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e"} Sep 16 19:04:33.522778 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:33.522747 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62"] Sep 16 19:04:33.523195 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:33.522957 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" containerID="cri-o://5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e" gracePeriod=30 Sep 16 19:04:36.652362 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.652342 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62_b5da5146-5864-4e91-b48e-57f372ad3bd8/storage-initializer/1.log" Sep 16 19:04:36.652761 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.652745 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62_b5da5146-5864-4e91-b48e-57f372ad3bd8/storage-initializer/0.log" Sep 16 19:04:36.655106 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.655092 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:36.761124 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761062 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-rbac-proxy-sar-config\") pod \"b5da5146-5864-4e91-b48e-57f372ad3bd8\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " Sep 16 19:04:36.761124 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761122 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls\") pod \"b5da5146-5864-4e91-b48e-57f372ad3bd8\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " Sep 16 19:04:36.761260 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761152 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6mww\" (UniqueName: \"kubernetes.io/projected/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-api-access-t6mww\") pod \"b5da5146-5864-4e91-b48e-57f372ad3bd8\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " Sep 16 19:04:36.761260 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5da5146-5864-4e91-b48e-57f372ad3bd8-kserve-provision-location\") pod \"b5da5146-5864-4e91-b48e-57f372ad3bd8\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " Sep 16 19:04:36.761260 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761224 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-cabundle-cert\") pod \"b5da5146-5864-4e91-b48e-57f372ad3bd8\" (UID: \"b5da5146-5864-4e91-b48e-57f372ad3bd8\") " Sep 16 19:04:36.761520 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761498 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5da5146-5864-4e91-b48e-57f372ad3bd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5da5146-5864-4e91-b48e-57f372ad3bd8" (UID: "b5da5146-5864-4e91-b48e-57f372ad3bd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:04:36.761567 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761508 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5da5146-5864-4e91-b48e-57f372ad3bd8" (UID: "b5da5146-5864-4e91-b48e-57f372ad3bd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:36.761631 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.761612 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5da5146-5864-4e91-b48e-57f372ad3bd8" (UID: "b5da5146-5864-4e91-b48e-57f372ad3bd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:36.763333 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.763310 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5da5146-5864-4e91-b48e-57f372ad3bd8" (UID: "b5da5146-5864-4e91-b48e-57f372ad3bd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:04:36.763468 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.763356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-api-access-t6mww" (OuterVolumeSpecName: "kube-api-access-t6mww") pod "b5da5146-5864-4e91-b48e-57f372ad3bd8" (UID: "b5da5146-5864-4e91-b48e-57f372ad3bd8"). InnerVolumeSpecName "kube-api-access-t6mww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:04:36.862246 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.862216 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mww\" (UniqueName: \"kubernetes.io/projected/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-api-access-t6mww\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:36.862246 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.862237 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5da5146-5864-4e91-b48e-57f372ad3bd8-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:36.862246 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.862246 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:36.862437 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.862255 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5da5146-5864-4e91-b48e-57f372ad3bd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:36.862437 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:36.862269 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5da5146-5864-4e91-b48e-57f372ad3bd8-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:37.252210 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62_b5da5146-5864-4e91-b48e-57f372ad3bd8/storage-initializer/1.log" Sep 16 19:04:37.252624 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252610 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62_b5da5146-5864-4e91-b48e-57f372ad3bd8/storage-initializer/0.log" Sep 16 19:04:37.252698 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252644 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerID="5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e" exitCode=1 Sep 16 19:04:37.252698 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" event={"ID":"b5da5146-5864-4e91-b48e-57f372ad3bd8","Type":"ContainerDied","Data":"5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e"} Sep 16 19:04:37.252811 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252710 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" Sep 16 19:04:37.252811 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62" event={"ID":"b5da5146-5864-4e91-b48e-57f372ad3bd8","Type":"ContainerDied","Data":"775af00b209ff9a950f2fd3c2828c1274b6b74166ed0e236862f15be163242f4"} Sep 16 19:04:37.252811 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.252789 2580 scope.go:117] "RemoveContainer" containerID="5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e" Sep 16 19:04:37.261846 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.261832 2580 scope.go:117] "RemoveContainer" containerID="07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d" Sep 16 19:04:37.269182 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.269166 2580 scope.go:117] "RemoveContainer" containerID="5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e" Sep 16 19:04:37.269449 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:37.269429 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e\": container with ID starting with 5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e not found: ID does not exist" containerID="5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e" Sep 16 19:04:37.269514 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.269456 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e"} err="failed to get container status \"5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e\": rpc error: code = NotFound desc = could not find container \"5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e\": container with ID starting with 5d7b50bdbb4061c16ee9b585526f36a55b5c87f8f9887081256860646a79212e not found: ID does not exist" Sep 16 19:04:37.269514 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.269473 2580 scope.go:117] "RemoveContainer" containerID="07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d" Sep 16 19:04:37.269722 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:37.269697 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d\": container with ID starting with 07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d not found: ID does not exist" containerID="07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d" Sep 16 19:04:37.269807 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.269725 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d"} err="failed to get container status \"07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d\": rpc error: code = NotFound desc = could not find container \"07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d\": container with ID starting with 07b2acc21c5f62ded33fcb49d1be25438b04983d316e03f2627b5931fcc6511d not found: ID does not exist" Sep 16 19:04:37.290026 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.290001 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62"] Sep 16 19:04:37.294155 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.294135 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a6bd8f-predictor-7c4587dc7b-bhf62"] Sep 16 19:04:37.402054 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:37.402024 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" path="/var/lib/kubelet/pods/b5da5146-5864-4e91-b48e-57f372ad3bd8/volumes" Sep 16 19:04:38.802579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802550 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx"] Sep 16 19:04:38.802970 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802877 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" Sep 16 19:04:38.802970 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802888 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" Sep 16 19:04:38.802970 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802899 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" Sep 16 19:04:38.802970 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802905 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" Sep 16 19:04:38.802970 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802961 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" Sep 16 19:04:38.802970 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.802969 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5da5146-5864-4e91-b48e-57f372ad3bd8" containerName="storage-initializer" Sep 16 19:04:38.807753 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.807735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.812735 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.812713 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:04:38.812938 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.812923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:04:38.813974 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.813956 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-78f6a-kube-rbac-proxy-sar-config\"" Sep 16 19:04:38.817515 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.817497 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-78f6a-predictor-serving-cert\"" Sep 16 19:04:38.823731 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.823709 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx"] Sep 16 19:04:38.878101 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.878081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kserve-provision-location\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.878210 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.878107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-cabundle-cert\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.878210 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.878127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8gx\" (UniqueName: \"kubernetes.io/projected/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-api-access-nt8gx\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.878210 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.878150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/464fedcc-d3f9-45b1-9d4b-88899eb792dd-proxy-tls\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.878414 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.878241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.979395 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.979340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.979491 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.979418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kserve-provision-location\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.979491 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.979445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-cabundle-cert\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.979491 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.979466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8gx\" (UniqueName: \"kubernetes.io/projected/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-api-access-nt8gx\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.979671 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.979496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/464fedcc-d3f9-45b1-9d4b-88899eb792dd-proxy-tls\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.979816 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.979794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kserve-provision-location\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.980122 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.980100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.980171 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.980114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-cabundle-cert\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.982006 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.981989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/464fedcc-d3f9-45b1-9d4b-88899eb792dd-proxy-tls\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:38.994338 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:38.994309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8gx\" (UniqueName: \"kubernetes.io/projected/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-api-access-nt8gx\") pod \"raw-sklearn-78f6a-predictor-865f85ddb6-clfcx\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:39.118107 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:39.118056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:39.394835 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:39.394810 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx"] Sep 16 19:04:39.396899 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:04:39.396870 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464fedcc_d3f9_45b1_9d4b_88899eb792dd.slice/crio-8f847bd4250c2bb33a4bc311a3799c4b4a998e7bea84d218c120bb60b4795432 WatchSource:0}: Error finding container 8f847bd4250c2bb33a4bc311a3799c4b4a998e7bea84d218c120bb60b4795432: Status 404 returned error can't find the container with id 8f847bd4250c2bb33a4bc311a3799c4b4a998e7bea84d218c120bb60b4795432 Sep 16 19:04:40.267686 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:40.267651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerStarted","Data":"51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b"} Sep 16 19:04:40.268149 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:40.267718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerStarted","Data":"8f847bd4250c2bb33a4bc311a3799c4b4a998e7bea84d218c120bb60b4795432"} Sep 16 19:04:42.403487 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:42.403458 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx_82742154-de84-48be-99fc-59f5a30857d3/kserve-container/0.log" Sep 16 19:04:42.657480 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:42.657408 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx"] Sep 16 19:04:42.657634 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:42.657612 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kserve-container" containerID="cri-o://3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32" gracePeriod=30 Sep 16 19:04:42.657694 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:42.657647 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kube-rbac-proxy" containerID="cri-o://1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c" gracePeriod=30 Sep 16 19:04:42.903241 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:42.903214 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:04:43.011698 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.011622 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhqgc\" (UniqueName: \"kubernetes.io/projected/82742154-de84-48be-99fc-59f5a30857d3-kube-api-access-dhqgc\") pod \"82742154-de84-48be-99fc-59f5a30857d3\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " Sep 16 19:04:43.011698 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.011669 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82742154-de84-48be-99fc-59f5a30857d3-kube-rbac-proxy-sar-config\") pod \"82742154-de84-48be-99fc-59f5a30857d3\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " Sep 16 19:04:43.011917 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.011760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls\") pod \"82742154-de84-48be-99fc-59f5a30857d3\" (UID: \"82742154-de84-48be-99fc-59f5a30857d3\") " Sep 16 19:04:43.012119 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.012083 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82742154-de84-48be-99fc-59f5a30857d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82742154-de84-48be-99fc-59f5a30857d3" (UID: "82742154-de84-48be-99fc-59f5a30857d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:43.013867 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.013835 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82742154-de84-48be-99fc-59f5a30857d3-kube-api-access-dhqgc" (OuterVolumeSpecName: "kube-api-access-dhqgc") pod "82742154-de84-48be-99fc-59f5a30857d3" (UID: "82742154-de84-48be-99fc-59f5a30857d3"). InnerVolumeSpecName "kube-api-access-dhqgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:04:43.013964 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.013905 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82742154-de84-48be-99fc-59f5a30857d3" (UID: "82742154-de84-48be-99fc-59f5a30857d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:04:43.112997 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.112967 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82742154-de84-48be-99fc-59f5a30857d3-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:43.112997 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.112993 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqgc\" (UniqueName: \"kubernetes.io/projected/82742154-de84-48be-99fc-59f5a30857d3-kube-api-access-dhqgc\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:43.113142 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.113003 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82742154-de84-48be-99fc-59f5a30857d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:04:43.281665 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281604 2580 generic.go:356] "Generic (PLEG): container finished" podID="82742154-de84-48be-99fc-59f5a30857d3" containerID="1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c" exitCode=2 Sep 16 19:04:43.281665 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281629 2580 generic.go:356] "Generic (PLEG): container finished" podID="82742154-de84-48be-99fc-59f5a30857d3" containerID="3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32" exitCode=2 Sep 16 19:04:43.281665 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281651 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" Sep 16 19:04:43.281851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" event={"ID":"82742154-de84-48be-99fc-59f5a30857d3","Type":"ContainerDied","Data":"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c"} Sep 16 19:04:43.281851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" event={"ID":"82742154-de84-48be-99fc-59f5a30857d3","Type":"ContainerDied","Data":"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32"} Sep 16 19:04:43.281851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281781 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx" event={"ID":"82742154-de84-48be-99fc-59f5a30857d3","Type":"ContainerDied","Data":"629ebb28b44a8cf0502337f1246674ebf18f2dbe0db9a4b126d0ffa682ddd193"} Sep 16 19:04:43.281851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.281814 2580 scope.go:117] "RemoveContainer" containerID="1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c" Sep 16 19:04:43.283650 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.283626 2580 generic.go:356] "Generic (PLEG): container finished" podID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerID="51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b" exitCode=0 Sep 16 19:04:43.283742 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.283690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerDied","Data":"51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b"} Sep 16 19:04:43.290372 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.290356 2580 scope.go:117] "RemoveContainer" containerID="3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32" Sep 16 19:04:43.298595 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.298579 2580 scope.go:117] "RemoveContainer" containerID="1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c" Sep 16 19:04:43.298919 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:43.298854 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c\": container with ID starting with 1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c not found: ID does not exist" containerID="1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c" Sep 16 19:04:43.298919 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.298880 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c"} err="failed to get container status \"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c\": rpc error: code = NotFound desc = could not find container \"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c\": container with ID starting with 1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c not found: ID does not exist" Sep 16 19:04:43.298919 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.298895 2580 scope.go:117] "RemoveContainer" containerID="3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32" Sep 16 19:04:43.299123 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:04:43.299106 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32\": container with ID starting with 3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32 not found: ID does not exist" containerID="3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32" Sep 16 19:04:43.299190 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.299134 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32"} err="failed to get container status \"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32\": rpc error: code = NotFound desc = could not find container \"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32\": container with ID starting with 3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32 not found: ID does not exist" Sep 16 19:04:43.299190 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.299155 2580 scope.go:117] "RemoveContainer" containerID="1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c" Sep 16 19:04:43.299434 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.299413 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c"} err="failed to get container status \"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c\": rpc error: code = NotFound desc = could not find container \"1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c\": container with ID starting with 1c4d3fcc896ab4408cb1c051f147c84cf806475685b5f21888320bb3d4fd843c not found: ID does not exist" Sep 16 19:04:43.299500 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.299435 2580 scope.go:117] "RemoveContainer" containerID="3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32" Sep 16 19:04:43.299663 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.299648 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32"} err="failed to get container status \"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32\": rpc error: code = NotFound desc = could not find container \"3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32\": container with ID starting with 3507759e6621d6e8aa887edf734e7dca5056793f18e0a671945cfacb47488f32 not found: ID does not exist" Sep 16 19:04:43.322847 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.322826 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx"] Sep 16 19:04:43.326180 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.326160 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-87fdd-predictor-c49f9d8fc-tkdmx"] Sep 16 19:04:43.401154 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:43.401129 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82742154-de84-48be-99fc-59f5a30857d3" path="/var/lib/kubelet/pods/82742154-de84-48be-99fc-59f5a30857d3/volumes" Sep 16 19:04:44.290528 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:44.290492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerStarted","Data":"cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279"} Sep 16 19:04:44.290898 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:44.290557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerStarted","Data":"a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4"} Sep 16 19:04:44.290898 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:44.290746 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:44.316746 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:44.316693 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podStartSLOduration=2.731116927 podStartE2EDuration="6.31667891s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585561983s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:44.314618365 +0000 UTC m=+1147.437465895" watchObservedRunningTime="2026-09-16 19:04:44.31667891 +0000 UTC m=+1147.439526416" Sep 16 19:04:45.295336 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:45.295306 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:45.296620 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:45.296589 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:04:46.299580 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:46.299542 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:04:51.303792 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:51.303765 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:04:51.304248 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:04:51.304226 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:05:01.304286 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:01.304211 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:05:11.304614 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:11.304575 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:05:21.304223 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:21.304181 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:05:31.304767 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:31.304731 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:05:37.368559 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:37.368531 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:05:37.377923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:37.377900 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:05:41.304172 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:41.304134 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:05:51.305067 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:05:51.305028 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:06:01.304537 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:01.304506 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:06:08.903783 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:08.903753 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx"] Sep 16 19:06:08.904506 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:08.904029 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" containerID="cri-o://a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4" gracePeriod=30 Sep 16 19:06:08.904506 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:08.904029 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kube-rbac-proxy" containerID="cri-o://cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279" gracePeriod=30 Sep 16 19:06:09.650912 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:09.650881 2580 generic.go:356] "Generic (PLEG): container finished" podID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerID="cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279" exitCode=2 Sep 16 19:06:09.651079 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:09.650945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerDied","Data":"cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279"} Sep 16 19:06:11.300499 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:11.300456 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused" Sep 16 19:06:11.304802 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:11.304776 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:06:12.844298 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.843865 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:06:12.938579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938502 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt8gx\" (UniqueName: \"kubernetes.io/projected/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-api-access-nt8gx\") pod \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " Sep 16 19:06:12.938579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kserve-provision-location\") pod \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " Sep 16 19:06:12.938579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938568 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/464fedcc-d3f9-45b1-9d4b-88899eb792dd-proxy-tls\") pod \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " Sep 16 19:06:12.938840 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-cabundle-cert\") pod \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " Sep 16 19:06:12.938840 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938752 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-rbac-proxy-sar-config\") pod \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\" (UID: \"464fedcc-d3f9-45b1-9d4b-88899eb792dd\") " Sep 16 19:06:12.938840 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938806 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "464fedcc-d3f9-45b1-9d4b-88899eb792dd" (UID: "464fedcc-d3f9-45b1-9d4b-88899eb792dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:06:12.939009 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.938994 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:06:12.939077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.939054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "464fedcc-d3f9-45b1-9d4b-88899eb792dd" (UID: "464fedcc-d3f9-45b1-9d4b-88899eb792dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:06:12.939077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.939056 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "464fedcc-d3f9-45b1-9d4b-88899eb792dd" (UID: "464fedcc-d3f9-45b1-9d4b-88899eb792dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:06:12.940766 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.940747 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-api-access-nt8gx" (OuterVolumeSpecName: "kube-api-access-nt8gx") pod "464fedcc-d3f9-45b1-9d4b-88899eb792dd" (UID: "464fedcc-d3f9-45b1-9d4b-88899eb792dd"). InnerVolumeSpecName "kube-api-access-nt8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:06:12.940864 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:12.940776 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464fedcc-d3f9-45b1-9d4b-88899eb792dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "464fedcc-d3f9-45b1-9d4b-88899eb792dd" (UID: "464fedcc-d3f9-45b1-9d4b-88899eb792dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:06:13.039767 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.039743 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8gx\" (UniqueName: \"kubernetes.io/projected/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-api-access-nt8gx\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:06:13.039767 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.039763 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/464fedcc-d3f9-45b1-9d4b-88899eb792dd-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:06:13.039890 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.039773 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:06:13.039890 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.039781 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/464fedcc-d3f9-45b1-9d4b-88899eb792dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:06:13.669799 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.669723 2580 generic.go:356] "Generic (PLEG): container finished" podID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerID="a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4" exitCode=0 Sep 16 19:06:13.669799 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.669783 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" Sep 16 19:06:13.670000 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.669804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerDied","Data":"a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4"} Sep 16 19:06:13.670000 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.669858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx" event={"ID":"464fedcc-d3f9-45b1-9d4b-88899eb792dd","Type":"ContainerDied","Data":"8f847bd4250c2bb33a4bc311a3799c4b4a998e7bea84d218c120bb60b4795432"} Sep 16 19:06:13.670000 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.669882 2580 scope.go:117] "RemoveContainer" containerID="cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279" Sep 16 19:06:13.680926 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.680908 2580 scope.go:117] "RemoveContainer" containerID="a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4" Sep 16 19:06:13.688832 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.688814 2580 scope.go:117] "RemoveContainer" containerID="51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b" Sep 16 19:06:13.689939 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.689922 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx"] Sep 16 19:06:13.695240 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.695221 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78f6a-predictor-865f85ddb6-clfcx"] Sep 16 19:06:13.696602 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.696579 2580 scope.go:117] "RemoveContainer" containerID="cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279" Sep 16 19:06:13.696860 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:06:13.696843 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279\": container with ID starting with cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279 not found: ID does not exist" containerID="cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279" Sep 16 19:06:13.696910 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.696868 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279"} err="failed to get container status \"cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279\": rpc error: code = NotFound desc = could not find container \"cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279\": container with ID starting with cbabd425d07cc787ae22cf079a7c6ef7d7bd1a82407d29b8d951c1993c798279 not found: ID does not exist" Sep 16 19:06:13.696910 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.696885 2580 scope.go:117] "RemoveContainer" containerID="a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4" Sep 16 19:06:13.697106 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:06:13.697080 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4\": container with ID starting with a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4 not found: ID does not exist" containerID="a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4" Sep 16 19:06:13.697156 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.697114 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4"} err="failed to get container status \"a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4\": rpc error: code = NotFound desc = could not find container \"a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4\": container with ID starting with a48239e82423f91e46d9cc5c5679896eca25c936ef684e4abaa595b162b395e4 not found: ID does not exist" Sep 16 19:06:13.697156 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.697135 2580 scope.go:117] "RemoveContainer" containerID="51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b" Sep 16 19:06:13.697421 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:06:13.697399 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b\": container with ID starting with 51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b not found: ID does not exist" containerID="51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b" Sep 16 19:06:13.697497 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:13.697429 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b"} err="failed to get container status \"51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b\": rpc error: code = NotFound desc = could not find container \"51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b\": container with ID starting with 51eadc691409157e170313075c620741bc19a6d1348c6784270ddd2b53147f8b not found: ID does not exist" Sep 16 19:06:14.130744 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.130712 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp"] Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131021 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kube-rbac-proxy" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131031 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kube-rbac-proxy" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131039 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="storage-initializer" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131045 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="storage-initializer" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131055 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kserve-container" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131060 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kserve-container" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131071 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kube-rbac-proxy" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131078 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kube-rbac-proxy" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131090 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" Sep 16 19:06:14.131104 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131095 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" Sep 16 19:06:14.131447 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131150 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kube-rbac-proxy" Sep 16 19:06:14.131447 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131159 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kserve-container" Sep 16 19:06:14.131447 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131166 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" containerName="kube-rbac-proxy" Sep 16 19:06:14.131447 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.131171 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82742154-de84-48be-99fc-59f5a30857d3" containerName="kserve-container" Sep 16 19:06:14.135651 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.135633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.137832 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.137806 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-2271c-predictor-serving-cert\"" Sep 16 19:06:14.137930 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.137816 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:06:14.138424 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.138407 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-2271c-kube-rbac-proxy-sar-config\"" Sep 16 19:06:14.138512 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.138424 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\"" Sep 16 19:06:14.138512 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.138446 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:06:14.138512 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.138463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:06:14.138647 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.138588 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:06:14.145079 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.145057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp"] Sep 16 19:06:14.246087 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.246058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.246234 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.246097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-cabundle-cert\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.246234 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.246117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-proxy-tls\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.246234 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.246151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbx6l\" (UniqueName: \"kubernetes.io/projected/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-api-access-dbx6l\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.246234 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.246181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kserve-provision-location\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.346640 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.346614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-cabundle-cert\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.346838 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.346645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-proxy-tls\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.346838 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.346677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbx6l\" (UniqueName: \"kubernetes.io/projected/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-api-access-dbx6l\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.346838 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.346720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kserve-provision-location\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.346838 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.346804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.347176 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.347154 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kserve-provision-location\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.347474 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.347457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-cabundle-cert\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.347555 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.347458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.349146 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.349126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-proxy-tls\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.354840 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.354818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbx6l\" (UniqueName: \"kubernetes.io/projected/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-api-access-dbx6l\") pod \"raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.446146 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.446095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:14.718099 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:14.718078 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp"] Sep 16 19:06:14.719827 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:06:14.719798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae85566d_9ec9_43b1_a6ee_5fb91fdb90f7.slice/crio-c1afdb79e7b3e80b1c64774a69dae462d85175bb68fcd68fb1be453179c584d0 WatchSource:0}: Error finding container c1afdb79e7b3e80b1c64774a69dae462d85175bb68fcd68fb1be453179c584d0: Status 404 returned error can't find the container with id c1afdb79e7b3e80b1c64774a69dae462d85175bb68fcd68fb1be453179c584d0 Sep 16 19:06:15.403112 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:15.403082 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464fedcc-d3f9-45b1-9d4b-88899eb792dd" path="/var/lib/kubelet/pods/464fedcc-d3f9-45b1-9d4b-88899eb792dd/volumes" Sep 16 19:06:15.682708 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:15.682624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerStarted","Data":"c691190efce25b3f449b49bee9640845ac8498730b49a0d26c4508e4134e4703"} Sep 16 19:06:15.682708 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:15.682680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerStarted","Data":"c1afdb79e7b3e80b1c64774a69dae462d85175bb68fcd68fb1be453179c584d0"} Sep 16 19:06:18.702485 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:18.702455 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerID="c691190efce25b3f449b49bee9640845ac8498730b49a0d26c4508e4134e4703" exitCode=0 Sep 16 19:06:18.702805 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:18.702509 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerDied","Data":"c691190efce25b3f449b49bee9640845ac8498730b49a0d26c4508e4134e4703"} Sep 16 19:06:19.709930 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:19.709897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerStarted","Data":"acb703f18819147f8c57bcd1c1e1e25b2eae5615d44afc9eb738278a630181ba"} Sep 16 19:06:19.710307 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:19.709963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerStarted","Data":"d479727ac539b264be011f60d0f3b5e0ea8470295e779cf3500075da623664d8"} Sep 16 19:06:19.710307 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:19.710149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:19.735964 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:19.735920 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podStartSLOduration=1.908166453 podStartE2EDuration="5.735898576s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827732123s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:19.733551507 +0000 UTC m=+1242.856399011" watchObservedRunningTime="2026-09-16 19:06:19.735898576 +0000 UTC m=+1242.858746079" Sep 16 19:06:20.714858 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:20.714826 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:20.716101 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:20.716070 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:06:21.719707 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:21.719669 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:06:26.725493 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:26.725462 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:06:26.725984 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:26.725957 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:06:36.726213 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:36.726168 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:06:46.726452 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:46.726417 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:06:56.726029 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:06:56.725985 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:07:06.726013 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:06.725978 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:07:16.726579 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:16.726539 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:07:26.726529 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:26.726488 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:07:27.401401 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:27.401356 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:07:34.221153 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:34.221123 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp"] Sep 16 19:07:34.221681 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:34.221351 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" containerID="cri-o://d479727ac539b264be011f60d0f3b5e0ea8470295e779cf3500075da623664d8" gracePeriod=30 Sep 16 19:07:34.221681 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:34.221373 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kube-rbac-proxy" containerID="cri-o://acb703f18819147f8c57bcd1c1e1e25b2eae5615d44afc9eb738278a630181ba" gracePeriod=30 Sep 16 19:07:35.047955 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:35.047923 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerID="acb703f18819147f8c57bcd1c1e1e25b2eae5615d44afc9eb738278a630181ba" exitCode=2 Sep 16 19:07:35.048125 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:35.047992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerDied","Data":"acb703f18819147f8c57bcd1c1e1e25b2eae5615d44afc9eb738278a630181ba"} Sep 16 19:07:36.720692 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:36.720658 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused" Sep 16 19:07:37.398396 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:37.398351 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:07:38.064103 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.064072 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerID="d479727ac539b264be011f60d0f3b5e0ea8470295e779cf3500075da623664d8" exitCode=0 Sep 16 19:07:38.064648 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.064617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerDied","Data":"d479727ac539b264be011f60d0f3b5e0ea8470295e779cf3500075da623664d8"} Sep 16 19:07:38.065694 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.065429 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd"] Sep 16 19:07:38.069658 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.069639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.071852 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.071834 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Sep 16 19:07:38.071946 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.071895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\"" Sep 16 19:07:38.081217 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.081195 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:07:38.082528 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.082504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd"] Sep 16 19:07:38.153608 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.153576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.153743 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.153611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbf4h\" (UniqueName: \"kubernetes.io/projected/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-api-access-vbf4h\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.153743 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.153700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe316994-0735-43c8-b1ce-582ec1dd5535-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.153743 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.153728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.153875 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.153759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe316994-0735-43c8-b1ce-582ec1dd5535-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.254214 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254149 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-rbac-proxy-sar-config\") pod \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " Sep 16 19:07:38.254214 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kserve-provision-location\") pod \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " Sep 16 19:07:38.254214 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254204 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbx6l\" (UniqueName: \"kubernetes.io/projected/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-api-access-dbx6l\") pod \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " Sep 16 19:07:38.254503 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254220 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-proxy-tls\") pod \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " Sep 16 19:07:38.254503 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-cabundle-cert\") pod \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\" (UID: \"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7\") " Sep 16 19:07:38.254503 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.254503 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbf4h\" (UniqueName: \"kubernetes.io/projected/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-api-access-vbf4h\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.254720 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254540 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" (UID: "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:07:38.254720 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" (UID: "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:38.254720 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254633 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" (UID: "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:38.254884 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe316994-0735-43c8-b1ce-582ec1dd5535-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.254884 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.254884 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe316994-0735-43c8-b1ce-582ec1dd5535-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.255044 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254904 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:07:38.255044 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254920 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:07:38.255044 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.254936 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:07:38.255292 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.255220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe316994-0735-43c8-b1ce-582ec1dd5535-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.255441 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.255319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.255559 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.255539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.256762 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.256733 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" (UID: "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:38.256851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.256773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-api-access-dbx6l" (OuterVolumeSpecName: "kube-api-access-dbx6l") pod "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" (UID: "ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7"). InnerVolumeSpecName "kube-api-access-dbx6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:38.257716 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.257701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe316994-0735-43c8-b1ce-582ec1dd5535-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.262195 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.262171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbf4h\" (UniqueName: \"kubernetes.io/projected/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-api-access-vbf4h\") pod \"isvc-canary-create-predictor-58f6457999-zbjwd\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.355278 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.355249 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbx6l\" (UniqueName: \"kubernetes.io/projected/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-kube-api-access-dbx6l\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:07:38.355362 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.355277 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:07:38.388260 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.388237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:38.661313 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:38.661287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd"] Sep 16 19:07:38.663304 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:07:38.663280 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe316994_0735_43c8_b1ce_582ec1dd5535.slice/crio-e0cb2c5ef34bc7ee695c6c45a02f8266f7f5891ad5453577f10603ea5f1fda1a WatchSource:0}: Error finding container e0cb2c5ef34bc7ee695c6c45a02f8266f7f5891ad5453577f10603ea5f1fda1a: Status 404 returned error can't find the container with id e0cb2c5ef34bc7ee695c6c45a02f8266f7f5891ad5453577f10603ea5f1fda1a Sep 16 19:07:39.070682 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.070648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerStarted","Data":"59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f"} Sep 16 19:07:39.071107 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.070715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerStarted","Data":"e0cb2c5ef34bc7ee695c6c45a02f8266f7f5891ad5453577f10603ea5f1fda1a"} Sep 16 19:07:39.072801 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.072772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" event={"ID":"ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7","Type":"ContainerDied","Data":"c1afdb79e7b3e80b1c64774a69dae462d85175bb68fcd68fb1be453179c584d0"} Sep 16 19:07:39.072915 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.072812 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp" Sep 16 19:07:39.072915 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.072834 2580 scope.go:117] "RemoveContainer" containerID="acb703f18819147f8c57bcd1c1e1e25b2eae5615d44afc9eb738278a630181ba" Sep 16 19:07:39.081929 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.081912 2580 scope.go:117] "RemoveContainer" containerID="d479727ac539b264be011f60d0f3b5e0ea8470295e779cf3500075da623664d8" Sep 16 19:07:39.091311 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.091274 2580 scope.go:117] "RemoveContainer" containerID="c691190efce25b3f449b49bee9640845ac8498730b49a0d26c4508e4134e4703" Sep 16 19:07:39.105873 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.105844 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp"] Sep 16 19:07:39.112251 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.112230 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-2271c-predictor-d45667466-9pxdp"] Sep 16 19:07:39.402222 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:39.402152 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" path="/var/lib/kubelet/pods/ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7/volumes" Sep 16 19:07:43.091958 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:43.091924 2580 generic.go:356] "Generic (PLEG): container finished" podID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerID="59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f" exitCode=0 Sep 16 19:07:43.092314 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:43.092002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerDied","Data":"59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f"} Sep 16 19:07:44.098261 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:44.098228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerStarted","Data":"e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308"} Sep 16 19:07:44.098654 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:44.098283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerStarted","Data":"fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0"} Sep 16 19:07:44.098654 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:44.098477 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:44.120491 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:44.120450 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podStartSLOduration=2.253704169 podStartE2EDuration="6.120435974s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.866731805s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:44.119184322 +0000 UTC m=+1327.242031840" watchObservedRunningTime="2026-09-16 19:07:44.120435974 +0000 UTC m=+1327.243283480" Sep 16 19:07:45.103299 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:45.103263 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:45.104441 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:45.104413 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:07:46.106945 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:46.106907 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:07:51.111820 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:51.111793 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:07:51.112303 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:07:51.112273 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:08:01.113320 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:08:01.113236 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:08:11.112670 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:08:11.112631 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:08:21.112686 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:08:21.112651 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:08:31.112668 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:08:31.112630 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:08:41.112545 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:08:41.112506 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:08:51.112269 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:08:51.112234 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:01.113482 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:01.113453 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:09:08.183272 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:08.176920 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd"] Sep 16 19:09:08.183272 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:08.177637 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" containerID="cri-o://fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0" gracePeriod=30 Sep 16 19:09:08.183272 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:08.178090 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kube-rbac-proxy" containerID="cri-o://e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308" gracePeriod=30 Sep 16 19:09:08.467154 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:08.467090 2580 generic.go:356] "Generic (PLEG): container finished" podID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerID="e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308" exitCode=2 Sep 16 19:09:08.467448 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:08.467165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerDied","Data":"e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308"} Sep 16 19:09:11.107876 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:11.107835 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused" Sep 16 19:09:11.113205 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:11.113181 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:12.129946 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.129922 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:09:12.231328 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231269 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe316994-0735-43c8-b1ce-582ec1dd5535-proxy-tls\") pod \"fe316994-0735-43c8-b1ce-582ec1dd5535\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " Sep 16 19:09:12.231328 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231307 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbf4h\" (UniqueName: \"kubernetes.io/projected/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-api-access-vbf4h\") pod \"fe316994-0735-43c8-b1ce-582ec1dd5535\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " Sep 16 19:09:12.231547 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231401 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe316994-0735-43c8-b1ce-582ec1dd5535-kserve-provision-location\") pod \"fe316994-0735-43c8-b1ce-582ec1dd5535\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " Sep 16 19:09:12.231547 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231449 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-cabundle-cert\") pod \"fe316994-0735-43c8-b1ce-582ec1dd5535\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " Sep 16 19:09:12.231547 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231468 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-rbac-proxy-sar-config\") pod \"fe316994-0735-43c8-b1ce-582ec1dd5535\" (UID: \"fe316994-0735-43c8-b1ce-582ec1dd5535\") " Sep 16 19:09:12.231808 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231766 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe316994-0735-43c8-b1ce-582ec1dd5535-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe316994-0735-43c8-b1ce-582ec1dd5535" (UID: "fe316994-0735-43c8-b1ce-582ec1dd5535"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:09:12.231808 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231784 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe316994-0735-43c8-b1ce-582ec1dd5535" (UID: "fe316994-0735-43c8-b1ce-582ec1dd5535"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:12.232067 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.231841 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe316994-0735-43c8-b1ce-582ec1dd5535" (UID: "fe316994-0735-43c8-b1ce-582ec1dd5535"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:12.233604 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.233582 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-api-access-vbf4h" (OuterVolumeSpecName: "kube-api-access-vbf4h") pod "fe316994-0735-43c8-b1ce-582ec1dd5535" (UID: "fe316994-0735-43c8-b1ce-582ec1dd5535"). InnerVolumeSpecName "kube-api-access-vbf4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:09:12.233688 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.233619 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe316994-0735-43c8-b1ce-582ec1dd5535-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe316994-0735-43c8-b1ce-582ec1dd5535" (UID: "fe316994-0735-43c8-b1ce-582ec1dd5535"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:09:12.332262 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.332234 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe316994-0735-43c8-b1ce-582ec1dd5535-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:09:12.332262 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.332261 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:09:12.332428 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.332270 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:09:12.332428 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.332279 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe316994-0735-43c8-b1ce-582ec1dd5535-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:09:12.332428 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.332288 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbf4h\" (UniqueName: \"kubernetes.io/projected/fe316994-0735-43c8-b1ce-582ec1dd5535-kube-api-access-vbf4h\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:09:12.486954 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.486887 2580 generic.go:356] "Generic (PLEG): container finished" podID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerID="fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0" exitCode=0 Sep 16 19:09:12.486954 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.486936 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerDied","Data":"fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0"} Sep 16 19:09:12.486954 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.486939 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" Sep 16 19:09:12.487137 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.486975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd" event={"ID":"fe316994-0735-43c8-b1ce-582ec1dd5535","Type":"ContainerDied","Data":"e0cb2c5ef34bc7ee695c6c45a02f8266f7f5891ad5453577f10603ea5f1fda1a"} Sep 16 19:09:12.487137 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.487000 2580 scope.go:117] "RemoveContainer" containerID="e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308" Sep 16 19:09:12.496654 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.496630 2580 scope.go:117] "RemoveContainer" containerID="fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0" Sep 16 19:09:12.507275 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.507255 2580 scope.go:117] "RemoveContainer" containerID="59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f" Sep 16 19:09:12.512718 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.512695 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd"] Sep 16 19:09:12.517194 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.517178 2580 scope.go:117] "RemoveContainer" containerID="e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308" Sep 16 19:09:12.517507 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:09:12.517484 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308\": container with ID starting with e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308 not found: ID does not exist" containerID="e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308" Sep 16 19:09:12.517571 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.517515 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308"} err="failed to get container status \"e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308\": rpc error: code = NotFound desc = could not find container \"e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308\": container with ID starting with e924ed71ce1ddc5eb46b43e7773cec6348f2dcfd1cff49e630a2d2b8b00be308 not found: ID does not exist" Sep 16 19:09:12.517571 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.517531 2580 scope.go:117] "RemoveContainer" containerID="fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0" Sep 16 19:09:12.517801 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:09:12.517778 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0\": container with ID starting with fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0 not found: ID does not exist" containerID="fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0" Sep 16 19:09:12.517851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.517811 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0"} err="failed to get container status \"fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0\": rpc error: code = NotFound desc = could not find container \"fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0\": container with ID starting with fce9bc37613759048b8ebbb6a078f177213245518df33816385f73b084a99bf0 not found: ID does not exist" Sep 16 19:09:12.517851 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.517835 2580 scope.go:117] "RemoveContainer" containerID="59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f" Sep 16 19:09:12.518153 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:09:12.518127 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f\": container with ID starting with 59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f not found: ID does not exist" containerID="59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f" Sep 16 19:09:12.518153 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.518145 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f"} err="failed to get container status \"59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f\": rpc error: code = NotFound desc = could not find container \"59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f\": container with ID starting with 59f74a1ad91231480bdfc4299882b1c76a96ec1efe742810edc93d1a4d95b08f not found: ID does not exist" Sep 16 19:09:12.520495 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:12.520475 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-zbjwd"] Sep 16 19:09:13.323547 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323508 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr"] Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323855 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="storage-initializer" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323866 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="storage-initializer" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323874 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="storage-initializer" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323880 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="storage-initializer" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323889 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323895 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323913 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kube-rbac-proxy" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323918 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kube-rbac-proxy" Sep 16 19:09:13.323923 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323926 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kube-rbac-proxy" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323931 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kube-rbac-proxy" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323956 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.323961 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.324013 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kserve-container" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.324024 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kube-rbac-proxy" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.324033 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" containerName="kube-rbac-proxy" Sep 16 19:09:13.324202 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.324043 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae85566d-9ec9-43b1-a6ee-5fb91fdb90f7" containerName="kserve-container" Sep 16 19:09:13.328675 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.328654 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.331527 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.331499 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:09:13.331527 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.331525 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:09:13.331911 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.331883 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:09:13.331911 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.331892 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:09:13.332055 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.331931 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Sep 16 19:09:13.332055 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.331986 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\"" Sep 16 19:09:13.332225 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.332211 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Sep 16 19:09:13.339176 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.339154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr"] Sep 16 19:09:13.389873 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.389844 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5"] Sep 16 19:09:13.398108 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.398081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.400175 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.400153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\"" Sep 16 19:09:13.406148 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.406120 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe316994-0735-43c8-b1ce-582ec1dd5535" path="/var/lib/kubelet/pods/fe316994-0735-43c8-b1ce-582ec1dd5535/volumes" Sep 16 19:09:13.406523 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.406507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5"] Sep 16 19:09:13.439819 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.439798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.439936 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.439845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.439994 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.439959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.440048 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.439998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.440048 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.440017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8d5\" (UniqueName: \"kubernetes.io/projected/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-api-access-gb8d5\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.540917 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.540893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541031 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.540929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.541031 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.540953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541121 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.541121 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541070 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.541197 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541235 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7d1440-f1bd-439a-afc1-5ef154d2c613-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.541278 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541278 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdp4\" (UniqueName: \"kubernetes.io/projected/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-api-access-xsdp4\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.541364 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8d5\" (UniqueName: \"kubernetes.io/projected/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-api-access-gb8d5\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541581 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541689 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.541784 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.541764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.543471 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.543454 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.548741 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.548720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8d5\" (UniqueName: \"kubernetes.io/projected/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-api-access-gb8d5\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.640526 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.640476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:13.641721 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.641702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdp4\" (UniqueName: \"kubernetes.io/projected/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-api-access-xsdp4\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.641779 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.641740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.641883 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.641865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.641928 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.641907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.641978 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.641962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7d1440-f1bd-439a-afc1-5ef154d2c613-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.642258 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.642238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.642437 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.642419 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.642506 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.642494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.644297 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.644277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7d1440-f1bd-439a-afc1-5ef154d2c613-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.650850 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.650831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdp4\" (UniqueName: \"kubernetes.io/projected/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-api-access-xsdp4\") pod \"isvc-canary-promote-predictor-5c9955746c-snhr5\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.709732 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.709697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:13.925594 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.925529 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr"] Sep 16 19:09:13.928328 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:09:13.928299 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8724fd_b6b2_41b9_8eaa_9dabc32f2fb1.slice/crio-537f1524758f862ca7350170f6c31b63456556d2259431947b6dcf451a806329 WatchSource:0}: Error finding container 537f1524758f862ca7350170f6c31b63456556d2259431947b6dcf451a806329: Status 404 returned error can't find the container with id 537f1524758f862ca7350170f6c31b63456556d2259431947b6dcf451a806329 Sep 16 19:09:13.983910 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:13.983878 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5"] Sep 16 19:09:13.985552 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:09:13.985525 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7d1440_f1bd_439a_afc1_5ef154d2c613.slice/crio-79521fe2d2d7358900331946d12d423af2e53a532664f593508fa878ddd37bd7 WatchSource:0}: Error finding container 79521fe2d2d7358900331946d12d423af2e53a532664f593508fa878ddd37bd7: Status 404 returned error can't find the container with id 79521fe2d2d7358900331946d12d423af2e53a532664f593508fa878ddd37bd7 Sep 16 19:09:14.501510 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:14.501463 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerStarted","Data":"6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257"} Sep 16 19:09:14.501993 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:14.501531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerStarted","Data":"79521fe2d2d7358900331946d12d423af2e53a532664f593508fa878ddd37bd7"} Sep 16 19:09:14.503169 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:14.503141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerStarted","Data":"6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592"} Sep 16 19:09:14.503309 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:14.503198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerStarted","Data":"537f1524758f862ca7350170f6c31b63456556d2259431947b6dcf451a806329"} Sep 16 19:09:18.522467 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:18.522427 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerID="6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257" exitCode=0 Sep 16 19:09:18.522876 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:18.522500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerDied","Data":"6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257"} Sep 16 19:09:18.524217 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:18.524198 2580 generic.go:356] "Generic (PLEG): container finished" podID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerID="6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592" exitCode=0 Sep 16 19:09:18.524290 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:18.524246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerDied","Data":"6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592"} Sep 16 19:09:19.530453 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.530414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerStarted","Data":"67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be"} Sep 16 19:09:19.530894 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.530481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerStarted","Data":"e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465"} Sep 16 19:09:19.530894 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.530780 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:19.531008 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.530933 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:19.532515 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.532481 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:09:19.532931 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.532912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerStarted","Data":"c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4"} Sep 16 19:09:19.532990 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.532957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerStarted","Data":"dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123"} Sep 16 19:09:19.533208 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.533192 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:19.533261 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.533242 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:19.534212 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.534190 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:09:19.574900 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.574860 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podStartSLOduration=2.545009707 podStartE2EDuration="6.574848575s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.029838868s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:19.574018321 +0000 UTC m=+1422.696865827" watchObservedRunningTime="2026-09-16 19:09:19.574848575 +0000 UTC m=+1422.697696156" Sep 16 19:09:19.612151 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:19.612110 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podStartSLOduration=2.746988986 podStartE2EDuration="6.612093325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865104339s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:19.608785597 +0000 UTC m=+1422.731633103" watchObservedRunningTime="2026-09-16 19:09:19.612093325 +0000 UTC m=+1422.734940832" Sep 16 19:09:20.537810 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:20.537767 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:09:20.538209 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:20.537894 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:09:25.542727 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:25.542700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:09:25.543087 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:25.542784 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:09:25.543290 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:25.543254 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:09:25.543577 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:25.543547 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:09:35.543833 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:35.543791 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:09:35.544218 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:35.543791 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:09:45.543857 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:45.543815 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:09:45.544456 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:45.543816 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:09:55.543929 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:55.543887 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:09:55.544346 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:09:55.543886 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:10:05.543490 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:05.543453 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:05.543902 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:05.543455 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:10:15.544262 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:15.544226 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:15.544727 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:15.544226 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:10:25.543745 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:25.543711 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:25.544246 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:25.543710 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:10:29.401084 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.401059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:10:29.575453 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.575421 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5"] Sep 16 19:10:29.575692 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.575644 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" containerID="cri-o://dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123" gracePeriod=30 Sep 16 19:10:29.575833 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.575714 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kube-rbac-proxy" containerID="cri-o://c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4" gracePeriod=30 Sep 16 19:10:29.576595 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.576569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:10:29.851002 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.850974 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerID="c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4" exitCode=2 Sep 16 19:10:29.851146 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:29.851051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerDied","Data":"c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4"} Sep 16 19:10:30.538000 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:30.537955 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused" Sep 16 19:10:33.421273 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.421250 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:10:33.535373 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535301 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-cabundle-cert\") pod \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " Sep 16 19:10:33.535373 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535338 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kserve-provision-location\") pod \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " Sep 16 19:10:33.535664 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535408 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-rbac-proxy-sar-config\") pod \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " Sep 16 19:10:33.535664 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535428 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7d1440-f1bd-439a-afc1-5ef154d2c613-proxy-tls\") pod \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " Sep 16 19:10:33.535664 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535445 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsdp4\" (UniqueName: \"kubernetes.io/projected/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-api-access-xsdp4\") pod \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\" (UID: \"ef7d1440-f1bd-439a-afc1-5ef154d2c613\") " Sep 16 19:10:33.535803 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535722 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef7d1440-f1bd-439a-afc1-5ef154d2c613" (UID: "ef7d1440-f1bd-439a-afc1-5ef154d2c613"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:10:33.535803 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef7d1440-f1bd-439a-afc1-5ef154d2c613" (UID: "ef7d1440-f1bd-439a-afc1-5ef154d2c613"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:10:33.535885 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.535798 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef7d1440-f1bd-439a-afc1-5ef154d2c613" (UID: "ef7d1440-f1bd-439a-afc1-5ef154d2c613"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:10:33.537546 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.537525 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-api-access-xsdp4" (OuterVolumeSpecName: "kube-api-access-xsdp4") pod "ef7d1440-f1bd-439a-afc1-5ef154d2c613" (UID: "ef7d1440-f1bd-439a-afc1-5ef154d2c613"). InnerVolumeSpecName "kube-api-access-xsdp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:10:33.537727 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.537710 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7d1440-f1bd-439a-afc1-5ef154d2c613-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef7d1440-f1bd-439a-afc1-5ef154d2c613" (UID: "ef7d1440-f1bd-439a-afc1-5ef154d2c613"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:10:33.636721 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.636699 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:10:33.636721 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.636718 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:10:33.636849 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.636728 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:10:33.636849 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.636737 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7d1440-f1bd-439a-afc1-5ef154d2c613-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:10:33.636849 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.636746 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdp4\" (UniqueName: \"kubernetes.io/projected/ef7d1440-f1bd-439a-afc1-5ef154d2c613-kube-api-access-xsdp4\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:10:33.871302 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.871271 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerID="dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123" exitCode=0 Sep 16 19:10:33.871454 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.871318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerDied","Data":"dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123"} Sep 16 19:10:33.871454 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.871335 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" Sep 16 19:10:33.871454 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.871405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5" event={"ID":"ef7d1440-f1bd-439a-afc1-5ef154d2c613","Type":"ContainerDied","Data":"79521fe2d2d7358900331946d12d423af2e53a532664f593508fa878ddd37bd7"} Sep 16 19:10:33.871454 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.871433 2580 scope.go:117] "RemoveContainer" containerID="c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4" Sep 16 19:10:33.880042 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.880027 2580 scope.go:117] "RemoveContainer" containerID="dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123" Sep 16 19:10:33.887304 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.887288 2580 scope.go:117] "RemoveContainer" containerID="6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257" Sep 16 19:10:33.894736 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.894721 2580 scope.go:117] "RemoveContainer" containerID="c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4" Sep 16 19:10:33.895004 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:10:33.894983 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4\": container with ID starting with c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4 not found: ID does not exist" containerID="c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4" Sep 16 19:10:33.895074 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.895011 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4"} err="failed to get container status \"c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4\": rpc error: code = NotFound desc = could not find container \"c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4\": container with ID starting with c37357e0edc8e71ab5e80ce204d3299377567c0b1971473572a9cc74dd391ac4 not found: ID does not exist" Sep 16 19:10:33.895074 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.895026 2580 scope.go:117] "RemoveContainer" containerID="dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123" Sep 16 19:10:33.895196 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.895178 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5"] Sep 16 19:10:33.895307 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:10:33.895288 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123\": container with ID starting with dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123 not found: ID does not exist" containerID="dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123" Sep 16 19:10:33.895346 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.895314 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123"} err="failed to get container status \"dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123\": rpc error: code = NotFound desc = could not find container \"dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123\": container with ID starting with dddd76dd5a04672ed9d489df6e78c6bd26303e21e303064e77f6ac3caf71d123 not found: ID does not exist" Sep 16 19:10:33.895346 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.895328 2580 scope.go:117] "RemoveContainer" containerID="6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257" Sep 16 19:10:33.895586 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:10:33.895572 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257\": container with ID starting with 6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257 not found: ID does not exist" containerID="6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257" Sep 16 19:10:33.895633 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.895591 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257"} err="failed to get container status \"6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257\": rpc error: code = NotFound desc = could not find container \"6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257\": container with ID starting with 6a376f756e76ec7742e13dfdf533f53e8d8f9ba65ada0bd00d5ea7318ccbb257 not found: ID does not exist" Sep 16 19:10:33.899076 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:33.899058 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-snhr5"] Sep 16 19:10:35.401640 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:35.401607 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" path="/var/lib/kubelet/pods/ef7d1440-f1bd-439a-afc1-5ef154d2c613/volumes" Sep 16 19:10:37.405675 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:37.405649 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:10:37.413623 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:10:37.413599 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:12:13.724303 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:13.721712 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr"] Sep 16 19:12:13.724303 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:13.722123 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" containerID="cri-o://e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465" gracePeriod=30 Sep 16 19:12:13.724303 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:13.722416 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kube-rbac-proxy" containerID="cri-o://67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be" gracePeriod=30 Sep 16 19:12:14.296831 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:14.296802 2580 generic.go:356] "Generic (PLEG): container finished" podID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerID="67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be" exitCode=2 Sep 16 19:12:14.296996 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:14.296843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerDied","Data":"67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be"} Sep 16 19:12:15.538281 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:15.538237 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused" Sep 16 19:12:17.777796 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.777770 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:12:17.875828 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.875768 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-rbac-proxy-sar-config\") pod \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " Sep 16 19:12:17.875828 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.875818 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb8d5\" (UniqueName: \"kubernetes.io/projected/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-api-access-gb8d5\") pod \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " Sep 16 19:12:17.875973 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.875848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kserve-provision-location\") pod \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " Sep 16 19:12:17.875973 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.875872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-proxy-tls\") pod \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " Sep 16 19:12:17.875973 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.875941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-cabundle-cert\") pod \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\" (UID: \"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1\") " Sep 16 19:12:17.876148 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.876123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" (UID: "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:12:17.876148 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.876135 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" (UID: "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:12:17.876374 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.876348 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" (UID: "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:12:17.877976 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.877957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-api-access-gb8d5" (OuterVolumeSpecName: "kube-api-access-gb8d5") pod "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" (UID: "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1"). InnerVolumeSpecName "kube-api-access-gb8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:12:17.878043 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.878027 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" (UID: "7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:12:17.976895 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.976870 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb8d5\" (UniqueName: \"kubernetes.io/projected/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-api-access-gb8d5\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:12:17.976895 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.976889 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kserve-provision-location\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:12:17.976895 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.976898 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:12:17.977058 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.976908 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-cabundle-cert\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:12:17.977058 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:17.976918 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:12:18.316576 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.316552 2580 generic.go:356] "Generic (PLEG): container finished" podID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerID="e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465" exitCode=0 Sep 16 19:12:18.316689 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.316606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerDied","Data":"e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465"} Sep 16 19:12:18.316689 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.316610 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" Sep 16 19:12:18.316689 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.316654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr" event={"ID":"7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1","Type":"ContainerDied","Data":"537f1524758f862ca7350170f6c31b63456556d2259431947b6dcf451a806329"} Sep 16 19:12:18.316689 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.316682 2580 scope.go:117] "RemoveContainer" containerID="67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be" Sep 16 19:12:18.325415 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.325400 2580 scope.go:117] "RemoveContainer" containerID="e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465" Sep 16 19:12:18.332559 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.332542 2580 scope.go:117] "RemoveContainer" containerID="6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592" Sep 16 19:12:18.339347 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.339329 2580 scope.go:117] "RemoveContainer" containerID="67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be" Sep 16 19:12:18.339672 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:12:18.339649 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be\": container with ID starting with 67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be not found: ID does not exist" containerID="67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be" Sep 16 19:12:18.339746 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.339694 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be"} err="failed to get container status \"67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be\": rpc error: code = NotFound desc = could not find container \"67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be\": container with ID starting with 67f60ae4cb52b944800bd8c1a64488d4c625c35954dbfa310b74fe8e45adb4be not found: ID does not exist" Sep 16 19:12:18.339746 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.339711 2580 scope.go:117] "RemoveContainer" containerID="e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465" Sep 16 19:12:18.339982 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:12:18.339962 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465\": container with ID starting with e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465 not found: ID does not exist" containerID="e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465" Sep 16 19:12:18.340042 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.339985 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465"} err="failed to get container status \"e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465\": rpc error: code = NotFound desc = could not find container \"e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465\": container with ID starting with e4271f2cd1d2efa318e41eb837044b69b709497a7b52e716e401922694d1f465 not found: ID does not exist" Sep 16 19:12:18.340042 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.340005 2580 scope.go:117] "RemoveContainer" containerID="6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592" Sep 16 19:12:18.340273 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:12:18.340254 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592\": container with ID starting with 6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592 not found: ID does not exist" containerID="6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592" Sep 16 19:12:18.340364 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.340278 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592"} err="failed to get container status \"6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592\": rpc error: code = NotFound desc = could not find container \"6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592\": container with ID starting with 6be25caa402c20dbfba692c66e721bd7e075c6993f27faffa116e9dff1b37592 not found: ID does not exist" Sep 16 19:12:18.342455 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.342436 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr"] Sep 16 19:12:18.346170 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:18.346150 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-sl4wr"] Sep 16 19:12:19.400756 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:19.400723 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" path="/var/lib/kubelet/pods/7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1/volumes" Sep 16 19:12:28.657545 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657474 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv"] Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657786 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657797 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657803 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657808 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657819 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="storage-initializer" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657826 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="storage-initializer" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657834 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="storage-initializer" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657839 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="storage-initializer" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657849 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kube-rbac-proxy" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657853 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kube-rbac-proxy" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657861 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kube-rbac-proxy" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657866 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kube-rbac-proxy" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657910 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kserve-container" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657919 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8724fd-b6b2-41b9-8eaa-9dabc32f2fb1" containerName="kube-rbac-proxy" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657926 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kserve-container" Sep 16 19:12:28.657980 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.657934 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7d1440-f1bd-439a-afc1-5ef154d2c613" containerName="kube-rbac-proxy" Sep 16 19:12:28.662414 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.662399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.673106 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.673080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:12:28.673800 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.673779 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\"" Sep 16 19:12:28.673908 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.673801 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\"" Sep 16 19:12:28.673908 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.673787 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:12:28.673908 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.673789 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\"" Sep 16 19:12:28.693593 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.693568 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv"] Sep 16 19:12:28.747483 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.747456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/360008bb-c570-4282-932b-be506f8f5168-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.747599 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.747512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rhl\" (UniqueName: \"kubernetes.io/projected/360008bb-c570-4282-932b-be506f8f5168-kube-api-access-79rhl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.747599 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.747581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/360008bb-c570-4282-932b-be506f8f5168-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.848740 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.848713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/360008bb-c570-4282-932b-be506f8f5168-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.848857 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.848758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rhl\" (UniqueName: \"kubernetes.io/projected/360008bb-c570-4282-932b-be506f8f5168-kube-api-access-79rhl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.848857 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.848782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/360008bb-c570-4282-932b-be506f8f5168-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.849399 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.849363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/360008bb-c570-4282-932b-be506f8f5168-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.851198 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.851180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/360008bb-c570-4282-932b-be506f8f5168-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.859029 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.859009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rhl\" (UniqueName: \"kubernetes.io/projected/360008bb-c570-4282-932b-be506f8f5168-kube-api-access-79rhl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-sbtnv\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:28.972520 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:28.972464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:29.261743 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:29.261719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv"] Sep 16 19:12:29.262701 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:12:29.262677 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360008bb_c570_4282_932b_be506f8f5168.slice/crio-f3156ce729a650955e3cc3506b213552aaf5af2d8ea99fd050c69e32ab6a44fa WatchSource:0}: Error finding container f3156ce729a650955e3cc3506b213552aaf5af2d8ea99fd050c69e32ab6a44fa: Status 404 returned error can't find the container with id f3156ce729a650955e3cc3506b213552aaf5af2d8ea99fd050c69e32ab6a44fa Sep 16 19:12:29.366627 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:29.366603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" event={"ID":"360008bb-c570-4282-932b-be506f8f5168","Type":"ContainerStarted","Data":"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078"} Sep 16 19:12:29.366734 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:29.366657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" event={"ID":"360008bb-c570-4282-932b-be506f8f5168","Type":"ContainerStarted","Data":"f3156ce729a650955e3cc3506b213552aaf5af2d8ea99fd050c69e32ab6a44fa"} Sep 16 19:12:30.372846 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:30.372809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" event={"ID":"360008bb-c570-4282-932b-be506f8f5168","Type":"ContainerStarted","Data":"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676"} Sep 16 19:12:30.373301 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:30.372941 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:30.403404 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:30.403333 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" podStartSLOduration=2.403319293 podStartE2EDuration="2.403319293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:30.401393588 +0000 UTC m=+1613.524241285" watchObservedRunningTime="2026-09-16 19:12:30.403319293 +0000 UTC m=+1613.526166799" Sep 16 19:12:31.377849 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:31.377820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:31.379504 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:31.379481 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:12:38.390060 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:12:38.390032 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:14:23.554120 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:23.554088 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-sbtnv_360008bb-c570-4282-932b-be506f8f5168/kserve-container/0.log" Sep 16 19:14:23.832344 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:23.832263 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv"] Sep 16 19:14:23.832581 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:23.832554 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kserve-container" containerID="cri-o://b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078" gracePeriod=30 Sep 16 19:14:23.832667 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:23.832571 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kube-rbac-proxy" containerID="cri-o://4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676" gracePeriod=30 Sep 16 19:14:24.068187 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.068165 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:14:24.160920 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.160856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rhl\" (UniqueName: \"kubernetes.io/projected/360008bb-c570-4282-932b-be506f8f5168-kube-api-access-79rhl\") pod \"360008bb-c570-4282-932b-be506f8f5168\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " Sep 16 19:14:24.160920 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.160895 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/360008bb-c570-4282-932b-be506f8f5168-proxy-tls\") pod \"360008bb-c570-4282-932b-be506f8f5168\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " Sep 16 19:14:24.161101 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.160926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/360008bb-c570-4282-932b-be506f8f5168-kube-rbac-proxy-sar-config\") pod \"360008bb-c570-4282-932b-be506f8f5168\" (UID: \"360008bb-c570-4282-932b-be506f8f5168\") " Sep 16 19:14:24.161302 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.161278 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360008bb-c570-4282-932b-be506f8f5168-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "360008bb-c570-4282-932b-be506f8f5168" (UID: "360008bb-c570-4282-932b-be506f8f5168"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:14:24.162960 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.162934 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360008bb-c570-4282-932b-be506f8f5168-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "360008bb-c570-4282-932b-be506f8f5168" (UID: "360008bb-c570-4282-932b-be506f8f5168"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:14:24.163080 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.163059 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360008bb-c570-4282-932b-be506f8f5168-kube-api-access-79rhl" (OuterVolumeSpecName: "kube-api-access-79rhl") pod "360008bb-c570-4282-932b-be506f8f5168" (UID: "360008bb-c570-4282-932b-be506f8f5168"). InnerVolumeSpecName "kube-api-access-79rhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:14:24.262043 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.262021 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rhl\" (UniqueName: \"kubernetes.io/projected/360008bb-c570-4282-932b-be506f8f5168-kube-api-access-79rhl\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:14:24.262043 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.262043 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/360008bb-c570-4282-932b-be506f8f5168-proxy-tls\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:14:24.262188 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.262054 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/360008bb-c570-4282-932b-be506f8f5168-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-157.ec2.internal\" DevicePath \"\"" Sep 16 19:14:24.874416 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874355 2580 generic.go:356] "Generic (PLEG): container finished" podID="360008bb-c570-4282-932b-be506f8f5168" containerID="4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676" exitCode=2 Sep 16 19:14:24.874416 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874406 2580 generic.go:356] "Generic (PLEG): container finished" podID="360008bb-c570-4282-932b-be506f8f5168" containerID="b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078" exitCode=2 Sep 16 19:14:24.874829 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874438 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" Sep 16 19:14:24.874829 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874437 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" event={"ID":"360008bb-c570-4282-932b-be506f8f5168","Type":"ContainerDied","Data":"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676"} Sep 16 19:14:24.874829 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" event={"ID":"360008bb-c570-4282-932b-be506f8f5168","Type":"ContainerDied","Data":"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078"} Sep 16 19:14:24.874829 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv" event={"ID":"360008bb-c570-4282-932b-be506f8f5168","Type":"ContainerDied","Data":"f3156ce729a650955e3cc3506b213552aaf5af2d8ea99fd050c69e32ab6a44fa"} Sep 16 19:14:24.874829 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.874572 2580 scope.go:117] "RemoveContainer" containerID="4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676" Sep 16 19:14:24.883122 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.883104 2580 scope.go:117] "RemoveContainer" containerID="b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078" Sep 16 19:14:24.890771 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.890755 2580 scope.go:117] "RemoveContainer" containerID="4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676" Sep 16 19:14:24.891021 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:14:24.891004 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676\": container with ID starting with 4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676 not found: ID does not exist" containerID="4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676" Sep 16 19:14:24.891077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891028 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676"} err="failed to get container status \"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676\": rpc error: code = NotFound desc = could not find container \"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676\": container with ID starting with 4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676 not found: ID does not exist" Sep 16 19:14:24.891077 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891043 2580 scope.go:117] "RemoveContainer" containerID="b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078" Sep 16 19:14:24.891247 ip-10-0-128-157 kubenswrapper[2580]: E0916 19:14:24.891232 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078\": container with ID starting with b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078 not found: ID does not exist" containerID="b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078" Sep 16 19:14:24.891293 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891247 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078"} err="failed to get container status \"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078\": rpc error: code = NotFound desc = could not find container \"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078\": container with ID starting with b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078 not found: ID does not exist" Sep 16 19:14:24.891293 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891258 2580 scope.go:117] "RemoveContainer" containerID="4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676" Sep 16 19:14:24.891488 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891469 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676"} err="failed to get container status \"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676\": rpc error: code = NotFound desc = could not find container \"4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676\": container with ID starting with 4e5fda61e3481bf2e69c881b2b190954874beabd751adb72f05e6e45b0ef4676 not found: ID does not exist" Sep 16 19:14:24.891541 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891490 2580 scope.go:117] "RemoveContainer" containerID="b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078" Sep 16 19:14:24.891698 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.891681 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078"} err="failed to get container status \"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078\": rpc error: code = NotFound desc = could not find container \"b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078\": container with ID starting with b48f2931cdf7af544cd23c4f4b76c79acf46eec44b87536a34923fe01c07d078 not found: ID does not exist" Sep 16 19:14:24.895365 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.895346 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv"] Sep 16 19:14:24.899414 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:24.899396 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-sbtnv"] Sep 16 19:14:25.402418 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:14:25.402366 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="360008bb-c570-4282-932b-be506f8f5168" path="/var/lib/kubelet/pods/360008bb-c570-4282-932b-be506f8f5168/volumes" Sep 16 19:15:00.431409 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:00.431361 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jxqsh_51e4870c-c32a-4cc1-b8c8-c0097c2f240c/global-pull-secret-syncer/0.log" Sep 16 19:15:00.517849 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:00.517819 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sgdnp_9922bad3-6630-46c7-8413-78a787ea7000/konnectivity-agent/0.log" Sep 16 19:15:00.582114 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:00.582094 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-157.ec2.internal_b301d645c57b142bcde84cc94cf8ddc5/haproxy/0.log" Sep 16 19:15:04.992056 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:04.992029 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-br5qf_21654739-4f9a-4dff-8bac-be315398da1b/node-exporter/0.log" Sep 16 19:15:05.019363 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.019340 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-br5qf_21654739-4f9a-4dff-8bac-be315398da1b/kube-rbac-proxy/0.log" Sep 16 19:15:05.049450 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.049434 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-br5qf_21654739-4f9a-4dff-8bac-be315398da1b/init-textfile/0.log" Sep 16 19:15:05.708407 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.708364 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5xmts_5eba91c7-e9c2-4e25-b894-85ceaeee050b/prometheus-operator/0.log" Sep 16 19:15:05.732612 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.732584 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5xmts_5eba91c7-e9c2-4e25-b894-85ceaeee050b/kube-rbac-proxy/0.log" Sep 16 19:15:05.764445 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.764422 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-8g8b8_07f271c7-4da0-46b9-8053-35e3f26a0ac3/prometheus-operator-admission-webhook/0.log" Sep 16 19:15:05.915262 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.915241 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ffbbd67d-vrc4d_f7f05bca-1cde-46b6-a40a-08b27aa00f4d/thanos-query/0.log" Sep 16 19:15:05.947942 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.947918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ffbbd67d-vrc4d_f7f05bca-1cde-46b6-a40a-08b27aa00f4d/kube-rbac-proxy-web/0.log" Sep 16 19:15:05.978745 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:05.978692 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ffbbd67d-vrc4d_f7f05bca-1cde-46b6-a40a-08b27aa00f4d/kube-rbac-proxy/0.log" Sep 16 19:15:06.008310 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:06.008292 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ffbbd67d-vrc4d_f7f05bca-1cde-46b6-a40a-08b27aa00f4d/prom-label-proxy/0.log" Sep 16 19:15:06.042313 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:06.042299 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ffbbd67d-vrc4d_f7f05bca-1cde-46b6-a40a-08b27aa00f4d/kube-rbac-proxy-rules/0.log" Sep 16 19:15:06.076446 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:06.076430 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ffbbd67d-vrc4d_f7f05bca-1cde-46b6-a40a-08b27aa00f4d/kube-rbac-proxy-metrics/0.log" Sep 16 19:15:07.504601 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:07.504577 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-zr89h_00a0b7b8-252a-4a58-86f6-0407fc0d3686/networking-console-plugin/0.log" Sep 16 19:15:08.008631 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.008603 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/1.log" Sep 16 19:15:08.012711 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.012691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ls8jm_7cf7deaa-e85f-4eda-980b-04cb0d77d071/console-operator/2.log" Sep 16 19:15:08.415201 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.415176 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-878585977-6vxkv_f1ce76f0-1313-40aa-ba05-2bba021a42a3/console/0.log" Sep 16 19:15:08.759687 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.759618 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c"] Sep 16 19:15:08.760120 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.760086 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kube-rbac-proxy" Sep 16 19:15:08.760120 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.760103 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kube-rbac-proxy" Sep 16 19:15:08.760120 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.760120 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kserve-container" Sep 16 19:15:08.760279 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.760129 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kserve-container" Sep 16 19:15:08.760279 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.760237 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kserve-container" Sep 16 19:15:08.760279 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.760249 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="360008bb-c570-4282-932b-be506f8f5168" containerName="kube-rbac-proxy" Sep 16 19:15:08.763152 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.763132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.765536 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.765519 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ng22b\"/\"kube-root-ca.crt\"" Sep 16 19:15:08.765936 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.765913 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ng22b\"/\"default-dockercfg-zthnt\"" Sep 16 19:15:08.765936 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.765926 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ng22b\"/\"openshift-service-ca.crt\"" Sep 16 19:15:08.775002 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.774982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c"] Sep 16 19:15:08.880254 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.880228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-lib-modules\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.880365 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.880278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-podres\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.880365 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.880323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-sys\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.880365 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.880339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-proc\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.880501 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.880397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6mb\" (UniqueName: \"kubernetes.io/projected/bd7b9832-a068-4395-9b91-b4e8f76fec02-kube-api-access-7b6mb\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981574 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6mb\" (UniqueName: \"kubernetes.io/projected/bd7b9832-a068-4395-9b91-b4e8f76fec02-kube-api-access-7b6mb\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981672 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-lib-modules\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981672 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-podres\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981744 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-sys\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981744 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-proc\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981868 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-lib-modules\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981868 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-podres\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981868 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-proc\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.981868 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.981826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd7b9832-a068-4395-9b91-b4e8f76fec02-sys\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:08.989648 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:08.989627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6mb\" (UniqueName: \"kubernetes.io/projected/bd7b9832-a068-4395-9b91-b4e8f76fec02-kube-api-access-7b6mb\") pod \"perf-node-gather-daemonset-nkh7c\" (UID: \"bd7b9832-a068-4395-9b91-b4e8f76fec02\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:09.048717 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:09.048693 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-gcrnl_4ef3f29c-6555-48e5-975f-9236bda75973/volume-data-source-validator/0.log" Sep 16 19:15:09.072323 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:09.072292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:09.343563 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:09.343360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c"] Sep 16 19:15:09.346083 ip-10-0-128-157 kubenswrapper[2580]: W0916 19:15:09.346058 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd7b9832_a068_4395_9b91_b4e8f76fec02.slice/crio-cc0caa29cdcbabfa5978ddd61983107368bfb855e8b269920732dc8146f75d93 WatchSource:0}: Error finding container cc0caa29cdcbabfa5978ddd61983107368bfb855e8b269920732dc8146f75d93: Status 404 returned error can't find the container with id cc0caa29cdcbabfa5978ddd61983107368bfb855e8b269920732dc8146f75d93 Sep 16 19:15:10.043981 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.043955 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bmgfk_1985d5f3-1e2f-46aa-8688-e2491814291f/dns/0.log" Sep 16 19:15:10.069661 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.069639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bmgfk_1985d5f3-1e2f-46aa-8688-e2491814291f/kube-rbac-proxy/0.log" Sep 16 19:15:10.070662 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.070638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" event={"ID":"bd7b9832-a068-4395-9b91-b4e8f76fec02","Type":"ContainerStarted","Data":"8ed93ebf84ec8bad8a14c468c39e6f895a2a8b8a44ee0443670cd35f50ef94ae"} Sep 16 19:15:10.070758 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.070690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" event={"ID":"bd7b9832-a068-4395-9b91-b4e8f76fec02","Type":"ContainerStarted","Data":"cc0caa29cdcbabfa5978ddd61983107368bfb855e8b269920732dc8146f75d93"} Sep 16 19:15:10.070813 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.070786 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:10.088786 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.088749 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" podStartSLOduration=2.088737636 podStartE2EDuration="2.088737636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:10.087496208 +0000 UTC m=+1773.210343718" watchObservedRunningTime="2026-09-16 19:15:10.088737636 +0000 UTC m=+1773.211585141" Sep 16 19:15:10.233196 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.233174 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vshfx_e50d547c-882d-4029-96ab-6d960ba6b3c1/dns-node-resolver/0.log" Sep 16 19:15:10.714343 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.714319 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-667c9bc59c-n25ng_b2259add-edc3-4ba5-b6c4-29d4d68c30a6/registry/0.log" Sep 16 19:15:10.733833 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:10.733815 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dw5h8_3d752bc5-c177-4081-8396-b249b8eacd3f/node-ca/0.log" Sep 16 19:15:11.937765 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:11.937733 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4sb7x_c7c1f908-5a84-4373-8039-cb6bbab6b949/serve-healthcheck-canary/0.log" Sep 16 19:15:12.371865 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:12.371840 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qdlfp_f9a248c6-76bc-457d-a2b8-c4b5231a04bb/insights-operator/0.log" Sep 16 19:15:12.372495 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:12.372479 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qdlfp_f9a248c6-76bc-457d-a2b8-c4b5231a04bb/insights-operator/1.log" Sep 16 19:15:12.393478 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:12.393461 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wdmf_9f2ac848-9cd2-42f2-97ae-74f25458627b/kube-rbac-proxy/0.log" Sep 16 19:15:12.417184 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:12.417166 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wdmf_9f2ac848-9cd2-42f2-97ae-74f25458627b/exporter/0.log" Sep 16 19:15:12.442813 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:12.442793 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wdmf_9f2ac848-9cd2-42f2-97ae-74f25458627b/extractor/0.log" Sep 16 19:15:15.073875 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:15.073846 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-764fc98f48-gz6dk_412944c9-5047-4e05-b933-447a7ddeefff/manager/0.log" Sep 16 19:15:15.155231 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:15.155205 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-4j46p_e5d95fff-1233-46ed-8a5a-842ff5ba8112/server/0.log" Sep 16 19:15:15.273310 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:15.273283 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-w8hq8_89a18619-da35-4341-8e1a-0dd02d8796ae/manager/0.log" Sep 16 19:15:15.386334 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:15.386231 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-x5swn_9096a269-a0b2-40a5-a4cf-94028071db62/seaweedfs/0.log" Sep 16 19:15:16.085295 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:16.085262 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-nkh7c" Sep 16 19:15:21.977140 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:21.977113 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/kube-multus-additional-cni-plugins/0.log" Sep 16 19:15:22.004897 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.004878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/egress-router-binary-copy/0.log" Sep 16 19:15:22.034282 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.034259 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/cni-plugins/0.log" Sep 16 19:15:22.063675 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.063657 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/bond-cni-plugin/0.log" Sep 16 19:15:22.089650 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.089628 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/routeoverride-cni/0.log" Sep 16 19:15:22.116230 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.116210 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/whereabouts-cni-bincopy/0.log" Sep 16 19:15:22.145932 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.145910 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pg56t_4cd8ecf3-a922-4d6e-a312-5a775bb163c0/whereabouts-cni/0.log" Sep 16 19:15:22.187687 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.187671 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gk2dp_adf41b5e-ddca-4667-903e-1c56061d678f/kube-multus/0.log" Sep 16 19:15:22.290551 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.290532 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8tzg8_593cdcf1-d960-4800-a58f-46e20476cb32/network-metrics-daemon/0.log" Sep 16 19:15:22.325886 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:22.325871 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8tzg8_593cdcf1-d960-4800-a58f-46e20476cb32/kube-rbac-proxy/0.log" Sep 16 19:15:23.650796 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.650762 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/ovn-controller/0.log" Sep 16 19:15:23.682520 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.682491 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/ovn-acl-logging/0.log" Sep 16 19:15:23.733173 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.733151 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/kube-rbac-proxy-node/0.log" Sep 16 19:15:23.770993 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.770927 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 19:15:23.805490 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.805465 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/northd/0.log" Sep 16 19:15:23.834439 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.834425 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/nbdb/0.log" Sep 16 19:15:23.860574 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.860556 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/sbdb/0.log" Sep 16 19:15:23.903247 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:23.903188 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gz585_6f03d949-aa22-4e8b-88d8-db70fd67f5b4/ovnkube-controller/0.log" Sep 16 19:15:25.188094 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:25.188065 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-tfj8x_edbdd95d-89d8-4559-aca5-cb5ba13283f9/check-endpoints/0.log" Sep 16 19:15:25.240755 ip-10-0-128-157 kubenswrapper[2580]: I0916 19:15:25.240732 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-t6x4g_b087b7c6-3e60-4ae6-a9a1-34cd06620b0b/network-check-target-container/0.log"