Sep 17 18:38:28.981800 ip-10-0-130-95 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:38:32.888701 ip-10-0-130-95 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:32.888701 ip-10-0-130-95 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:38:32.888701 ip-10-0-130-95 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:32.888701 ip-10-0-130-95 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:38:32.888701 ip-10-0-130-95 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:32.889476 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.888773    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:38:32.893857 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893834    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:32.893857 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893853    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:32.893857 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893858    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:32.893857 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893861    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:32.893857 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893865    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893869    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893872    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893875    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893879    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893882    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893885    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893888    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893891    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893894    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893897    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893900    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893902    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893906    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893909    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893912    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893915    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893918    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893921    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:32.894063 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893923    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893926    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893929    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893932    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893935    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893938    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893941    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893944    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893947    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893950    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893953    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893955    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893958    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893961    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893964    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893968    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893971    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893974    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893976    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893979    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:32.894574 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893983    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893985    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893988    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893991    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.893996    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894000    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894003    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894006    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894010    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894013    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894016    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894019    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894022    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894025    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894028    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894031    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894034    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894036    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894039    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:32.895090 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894042    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894045    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894049    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894053    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894056    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894059    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894062    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894066    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894070    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894073    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894077    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894080    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894083    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894086    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894088    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894091    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894094    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894097    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894100    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894104    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:32.895603 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894107    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894110    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894113    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894115    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894118    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894123    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894126    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894129    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894133    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894136    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894139    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894143    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894146    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894148    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894152    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894155    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894158    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894927    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894936    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894940    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:32.896151 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894944    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894948    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894951    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894954    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894957    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894960    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894963    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894966    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894969    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894972    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894975    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894978    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894981    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894983    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894986    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894989    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894992    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894996    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.894999    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:32.896681 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895002    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895005    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895008    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895011    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895014    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895017    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895020    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895023    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895026    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895029    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895032    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895037    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895041    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895045    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895049    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895053    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895057    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895060    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:32.897206 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895063    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895067    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895070    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895073    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895076    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895079    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895082    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895086    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895089    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895092    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895095    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895099    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895102    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895105    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895108    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895111    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895115    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895119    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895122    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:32.897711 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895125    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895128    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895131    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895135    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895138    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895141    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895145    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895148    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895152    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895155    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895158    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895161    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895164    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895167    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895170    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895173    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895175    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895179    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895182    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895184    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:32.898234 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895187    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895190    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895193    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895195    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895198    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895201    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895204    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895207    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895210    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895213    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895215    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895218    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895221    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895224    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895227    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895230    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895233    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895236    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895238    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895241    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:32.898794 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895629    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895635    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895638    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895641    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895644    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895647    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895650    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895653    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895656    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895659    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895662    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895664    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895667    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895670    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895673    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895676    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895679    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895682    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895685    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895688    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:32.899333 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895691    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895694    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895696    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895699    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895702    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895705    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895707    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895711    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895714    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895718    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895722    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895725    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895728    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895731    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895734    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895739    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895743    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895746    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895749    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:32.899868 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895752    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895755    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895758    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895761    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895764    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895767    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895770    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895772    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895775    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895778    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895781    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895783    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895786    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895789    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895792    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895795    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895798    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895800    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895804    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895807    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:32.900357 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895809    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895812    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895815    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895818    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895821    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895824    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895827    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895830    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895833    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895841    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895844    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895847    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895850    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895853    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895856    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895859    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895863    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895865    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895868    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895871    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:32.900902 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895874    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895877    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895881    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895885    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895888    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895892    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895895    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895898    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895900    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895906    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895909    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895911    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895914    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895917    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895920    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895923    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895926    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895929    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895931    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:32.901542 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.895934    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896017    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896063    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896071    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896075    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896079    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896083    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896087    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896091    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896095    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896098    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896102    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896105    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896109    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896112    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896115    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896118    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896121    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896124    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896129    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896132    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896136    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896141    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896145    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:38:32.902038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896149    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896153    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896156    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896160    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896164    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896167    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896170    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896173    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896179    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896183    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896186    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896189    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896193    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896197    2565 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896200    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896207    2565 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896210    2565 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896213    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896217    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896220    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896224    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896227    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896230    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896233    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896237    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:38:32.902686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896240    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896243    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896246    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896249    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896252    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896257    2565 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896261    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896264    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896268    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896271    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896274    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896278    2565 flags.go:64] FLAG: --help="false"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896281    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-130-95.ec2.internal"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896284    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896288    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896292    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896296    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896299    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896302    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896306    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896309    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896312    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896316    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896319    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:38:32.903403 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896322    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896325    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896328    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896331    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896335    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896338    2565 flags.go:64] FLAG: --lock-file=""
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896341    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896344    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896347    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896353    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896356    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896359    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896362    2565 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896368    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896372    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896375    2565 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896379    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896383    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896386    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896390    2565 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896394    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896397    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896400    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896405    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896408    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:38:32.904131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896411    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896415    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896422    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896425    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896429    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896433    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896436    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896442    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896445    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896449    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896452    2565 flags.go:64] FLAG: --port="10250"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896455    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896458    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04eaefacdc0cf6d4c"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896461    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896465    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896468    2565 flags.go:64] FLAG: --register-node="true"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896471    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896475    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896478    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896481    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896484    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896489    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896492    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896495    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896499    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:38:32.904880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896502    2565 flags.go:64] FLAG: --runonce="false"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896505    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896524    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896529    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896533    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896540    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896544    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896547    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896551    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896554    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896557    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896561    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896564    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896568    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896571    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896574    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896580    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896583    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896586    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896591    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896594    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896597    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896600    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896603    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896606    2565 flags.go:64] FLAG: --v="2"
Sep 17 18:38:32.905907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896610    2565 flags.go:64] FLAG: --version="false"
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896614    2565 flags.go:64] FLAG: --vmodule=""
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896618    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.896624    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896757    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896760    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896764    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896767    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896770    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896774    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896777    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896780    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896784    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896787    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896790    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896793    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896796    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896799    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896803    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896808    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:32.906723 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896813    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896817    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896820    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896823    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896826    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896830    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896834    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896838    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896841    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896844    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896847    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896850    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896853    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896856    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896858    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896863    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896866    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896868    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896872    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:32.907261 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896875    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896878    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896880    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896883    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896886    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896891    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896894    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896896    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896899    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896902    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896905    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896908    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896911    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896914    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896917    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896920    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896923    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896926    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896928    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896931    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:32.907797 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896934    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896936    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896939    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896942    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896944    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896947    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896950    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896953    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896957    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896960    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896964    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896967    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896970    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896972    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896975    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896978    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896981    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896985    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896988    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896991    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:32.908331 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896994    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896997    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.896999    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897002    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897005    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897008    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897011    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897014    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897017    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897020    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897023    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897025    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897028    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897031    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897034    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897037    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897039    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897042    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897045    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:32.908891 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897048    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897053    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897055    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897058    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.897061    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.897531    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.905614    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.905649    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.905677    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.905692    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905923    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905932    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905939    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905945    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905950    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:32.909411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905955    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905960    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905971    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905975    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905980    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905985    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905989    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905995    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.905999    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906004    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906009    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906015    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906020    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906024    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906037    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906043    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906049    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906054    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906059    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:32.909923 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906063    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906068    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906072    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906077    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906156    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906250    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906293    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906298    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906304    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906309    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906314    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906318    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906321    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906325    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906328    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906331    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906335    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906338    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906341    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906344    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:32.910429 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906347    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906350    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906353    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906355    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906359    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906361    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906364    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906368    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906371    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906374    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906377    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906379    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906382    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906386    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906391    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906396    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906399    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906403    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906407    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906410    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:32.910990 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906413    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906415    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906418    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906421    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906424    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906427    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906430    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906433    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906436    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906442    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906449    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906454    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906459    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906464    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906468    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906473    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906477    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906479    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906482    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:32.911585 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906485    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906488    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906491    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906494    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906497    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906500    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906502    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906505    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906510    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906524    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906528    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906531    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906535    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906540    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906544    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:32.912085 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906548    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.906557    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906705    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906711    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906714    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906719    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906724    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906728    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906734    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906738    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906742    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906745    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906748    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906750    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906753    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:32.912482 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906756    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906759    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906762    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906766    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906770    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906773    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906776    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906779    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906782    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906785    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906788    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906792    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906796    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906799    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906802    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906805    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906809    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906815    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906820    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906825    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:32.912896 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906828    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906831    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906834    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906836    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906839    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906842    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906844    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906847    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906850    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906853    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906856    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906858    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906863    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906866    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906869    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906871    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906874    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906877    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906880    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906883    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:32.913443 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906885    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906888    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906891    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906896    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906901    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906905    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906910    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906913    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906916    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906919    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906921    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906924    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906927    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906930    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906933    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906935    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906938    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906941    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906944    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906947    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:32.913972 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906950    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906952    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906955    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906958    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906961    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906964    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906967    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906971    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906975    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906979    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906984    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906988    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906993    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.906997    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907000    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907005    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907008    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907012    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:32.914522 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907016    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907019    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907022    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907025    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907028    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907031    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907034    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:32.907037    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.907042    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.907157    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.909590    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.910544    2565 server.go:1019] "Starting client certificate rotation"
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.910652    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:38:32.915000 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.910688    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:38:32.915404 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.913228    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:38:32.915404 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.915098    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:38:32.927185 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.927167    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:38:32.929163 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.929147    2565 log.go:25] "Validated CRI v1 image API"
Sep 17 18:38:32.930097 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.930070    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:38:32.932420 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.932398    2565 fs.go:135] Filesystem UUIDs: map[7875365c-ca70-4155-b42b-a49264f696bf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 971b0644-9708-4f1d-8bb6-9f8ba05af569:/dev/nvme0n1p3]
Sep 17 18:38:32.932490 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.932420    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:38:32.937277 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.937152    2565 manager.go:216] Machine: {Timestamp:2026-09-17 18:38:32.936652098 +0000 UTC m=+2.222964610 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098373 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217edf13d0869e19106971eb130da8 SystemUUID:ec217edf-13d0-869e-1910-6971eb130da8 BootID:9b28ea1d-b1a3-47e3-a2bb-5e54e06169f4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f0:4c:f0:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f0:4c:f0:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:fb:2e:34:c0:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:38:32.937277 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.937273    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:38:32.937447 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.937383    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:38:32.938987 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.938960    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:38:32.939192 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.939009    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:38:32.939885 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.939872    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:38:32.939942 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.939891    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:38:32.939942 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.939902    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:38:32.939942 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.939918    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:38:32.941644 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.941630    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:38:32.944962 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.944943    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:38:32.945777 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.945762    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:38:32.945915 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.945903    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:38:32.945979 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.945955    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:38:32.945979 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.945973    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:38:32.946075 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.945995    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:38:32.946075 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.946009    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:38:32.946162 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.946103    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:38:32.946650 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.946635    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:38:32.946735 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.946662    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:38:32.958611 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.958593    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:38:32.961138 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.961120    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:38:32.961232 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.961161    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:38:32.962991 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.962975    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.962997    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963007    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963014    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963027    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963035    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963055    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963064    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963075    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:38:32.963081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963084    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:38:32.963424 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963097    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:38:32.963424 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963106    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:38:32.963424 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963147    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:38:32.963424 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.963156    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:38:32.964746 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:32.964722    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:38:32.964746 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:32.964732    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:38:32.967158 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.967144    2565 server.go:1268] "Started kubelet"
Sep 17 18:38:32.967224 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.967189    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:38:32.967346 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.967294    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:38:32.967406 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.967365    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:38:32.967917 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.967880    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:38:32.967907 ip-10-0-130-95 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:38:32.968538 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.968479    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:38:32.968538 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.968483    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:38:32.971615 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:32.971588    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:38:32.976426 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.976407    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:38:32.976426 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.976417    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:38:32.976580 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:32.976493    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:32.976991 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.976973    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:38:32.977065 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977033    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:38:32.977065 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977000    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:38:32.977165 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977088    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:38:32.977165 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977095    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:38:32.977228 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977190    2565 factory.go:55] Registering systemd factory
Sep 17 18:38:32.977228 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977202    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:38:32.977413 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977400    2565 factory.go:153] Registering CRI-O factory
Sep 17 18:38:32.977413 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977412    2565 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:38:32.977530 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977482    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:38:32.977530 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977499    2565 factory.go:103] Registering Raw factory
Sep 17 18:38:32.977530 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977518    2565 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:38:32.977920 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:32.977909    2565 manager.go:349] Starting recovery of all containers
Sep 17 18:38:32.978847 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:32.978819    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:38:32.978983 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:32.978854    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:38:33.002643 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.002622    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phlzs"
Sep 17 18:38:33.003476 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.003461    2565 manager.go:354] Recovery completed
Sep 17 18:38:33.007325 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.007314    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.008423 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.008407    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phlzs"
Sep 17 18:38:33.009240 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009224    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.009298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009249    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.009298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009259    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.009752 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009736    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:38:33.009752 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009752    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:38:33.009854 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009769    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:38:33.009854 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.009739    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.010582 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.010568    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.010642 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.010590    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.010642 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.010599    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.012043 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.012029    2565 policy_none.go:49] "None policy: Start"
Sep 17 18:38:33.012119 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.012048    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:38:33.012119 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.012061    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:38:33.014372 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.014358    2565 policy_none.go:47] "Start"
Sep 17 18:38:33.015644 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.014779    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-95.ec2.internal.18d62faae5776e93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-95.ec2.internal,UID:ip-10-0-130-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-95.ec2.internal,},FirstTimestamp:2026-09-17 18:38:32.967106195 +0000 UTC m=+2.253418703,LastTimestamp:2026-09-17 18:38:32.967106195 +0000 UTC m=+2.253418703,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-95.ec2.internal,}"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053019    2565 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.053059    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053073    2565 server.go:87] "Starting device plugin registration server"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053323    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053346    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053463    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053577    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.053591    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.054147    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.054182    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.055183    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.056310    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.056352    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.056383    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.056421    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:38:33.061557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.058368    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:33.154417 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.154361    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.155099 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.155081    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.155184 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.155107    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.155184 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.155119    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.155184 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.155145    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.157244 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.157227    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"]
Sep 17 18:38:33.157293 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.157285    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.157896 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.157884    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.157946 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.157910    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.157946 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.157920    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.159880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.159868    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.159986 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.159973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.160022 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.159998    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.160512 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.160496    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.160609 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.160515    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.160609 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.160527    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.160609 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.160537    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.160609 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.160542    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.160609 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.160552    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.162567 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.162550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.162671 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.162576    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:33.163209 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.163192    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:33.163280 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.163216    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:33.163280 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.163229    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:33.164478 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.164461    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.164564 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.164484    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-95.ec2.internal\": node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.179959 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.179937    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-95.ec2.internal\" not found" node="ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.183297 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.183280    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-95.ec2.internal\" not found" node="ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.197057 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.197039    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.278563 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.278544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02b9657d18b7f38a78ec6983e42d11dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal\" (UID: \"02b9657d18b7f38a78ec6983e42d11dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.278621 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.278590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b9657d18b7f38a78ec6983e42d11dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal\" (UID: \"02b9657d18b7f38a78ec6983e42d11dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.278656 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.278626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f3ea4320296e328c2dbf6e5b180d445-config\") pod \"kube-apiserver-proxy-ip-10-0-130-95.ec2.internal\" (UID: \"8f3ea4320296e328c2dbf6e5b180d445\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.297613 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.297596    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.378952 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.378923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b9657d18b7f38a78ec6983e42d11dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal\" (UID: \"02b9657d18b7f38a78ec6983e42d11dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.379028 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.378957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f3ea4320296e328c2dbf6e5b180d445-config\") pod \"kube-apiserver-proxy-ip-10-0-130-95.ec2.internal\" (UID: \"8f3ea4320296e328c2dbf6e5b180d445\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.379028 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.378982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02b9657d18b7f38a78ec6983e42d11dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal\" (UID: \"02b9657d18b7f38a78ec6983e42d11dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.379028 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.379008    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b9657d18b7f38a78ec6983e42d11dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal\" (UID: \"02b9657d18b7f38a78ec6983e42d11dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.379115 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.379033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f3ea4320296e328c2dbf6e5b180d445-config\") pod \"kube-apiserver-proxy-ip-10-0-130-95.ec2.internal\" (UID: \"8f3ea4320296e328c2dbf6e5b180d445\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.379115 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.379010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02b9657d18b7f38a78ec6983e42d11dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal\" (UID: \"02b9657d18b7f38a78ec6983e42d11dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.398063 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.398031    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.484727 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.484707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.487866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.487853    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:33.498546 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.498529    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.598948 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.598927    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.699377 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.699357    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.799966 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.799916    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.900376 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:33.900351    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:33.910505 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.910488    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:38:33.910637 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.910621    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:38:33.991716 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:33.990816    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:34.001390 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.001367    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:34.014496 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.014469    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:33:33 +0000 UTC" deadline="2028-03-23 16:25:35.360691294 +0000 UTC"
Sep 17 18:38:34.014496 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.014495    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13269h47m1.346199898s"
Sep 17 18:38:34.047650 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.047630    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:34.049522 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.049496    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:34.049606 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.049531    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:34.049606 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.049543    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:34.101839 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.101822    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:34.166962 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:34.166934    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3ea4320296e328c2dbf6e5b180d445.slice/crio-192564d9be00684cb54bc65ffa377a87aae2910150c906df3af7aaa704e22a6f WatchSource:0}: Error finding container 192564d9be00684cb54bc65ffa377a87aae2910150c906df3af7aaa704e22a6f: Status 404 returned error can't find the container with id 192564d9be00684cb54bc65ffa377a87aae2910150c906df3af7aaa704e22a6f
Sep 17 18:38:34.167524 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:34.167504    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b9657d18b7f38a78ec6983e42d11dc.slice/crio-a978b4ea2fc2f66df9233140679d6cf8a3af51087f3f7608b524b980b5501ff3 WatchSource:0}: Error finding container a978b4ea2fc2f66df9233140679d6cf8a3af51087f3f7608b524b980b5501ff3: Status 404 returned error can't find the container with id a978b4ea2fc2f66df9233140679d6cf8a3af51087f3f7608b524b980b5501ff3
Sep 17 18:38:34.172699 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.172681    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:38:34.202893 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.202871    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:34.303374 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.303353    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:34.403820 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.403765    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:34.504528 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.504500    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-95.ec2.internal\" not found"
Sep 17 18:38:34.537933 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.537906    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:34.574105 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.574084    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:34.577093 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.577075    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:34.586445 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.586419    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:38:34.587304 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.587292    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal"
Sep 17 18:38:34.599467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.599450    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:38:34.948584 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.948557    2565 apiserver.go:52] "Watching apiserver"
Sep 17 18:38:34.955425 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.955401    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:38:34.955849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.955822    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9","openshift-dns/node-resolver-99thn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal","openshift-multus/multus-qs64h","openshift-multus/network-metrics-daemon-jhxpv","openshift-network-diagnostics/network-check-target-pzcln","openshift-network-operator/iptables-alerter-g4d9g","kube-system/global-pull-secret-syncer-gvpvp","kube-system/konnectivity-agent-vjx5b","openshift-cluster-node-tuning-operator/tuned-tzs2x","openshift-image-registry/node-ca-f8k74","openshift-multus/multus-additional-cni-plugins-2jnzg","openshift-ovn-kubernetes/ovnkube-node-2lf85"]
Sep 17 18:38:34.958215 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.958193    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:34.960582 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.960552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.961535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.961506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-blmv2\""
Sep 17 18:38:34.961535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.961526    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:38:34.961688 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.961519    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:38:34.962889 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.962839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:34.965279 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.965262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.966666 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.966471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:38:34.966666 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.966506    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:38:34.966666 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.966505    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.966666 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.966558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:38:34.967817 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.967797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.969165 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.969150    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-frt59\""
Sep 17 18:38:34.969689 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.969671    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:34.969777 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.969742    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:34.971366 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.971345    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:38:34.971523 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.971427    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:38:34.971523 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.971476    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.971631 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.971349    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.972139 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.971965    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:34.972139 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.972035    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:34.972273 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzsgn\""
Sep 17 18:38:34.972366 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:38:34.972469 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972401    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:38:34.972469 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972466    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:38:34.972574 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972403    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bqs6z\""
Sep 17 18:38:34.972619 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cv5k9\""
Sep 17 18:38:34.972697 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:38:34.972752 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.972732    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:38:34.973059 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.973041    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:38:34.981354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.975613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:34.981354 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:34.975707    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:34.981354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.978901    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:38:34.981354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.979770    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.982847 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.982828    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:34.985684 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.985665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:34.985785 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.985725    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-czbvz\""
Sep 17 18:38:34.986232 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:38:34.986297 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986269    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.986379 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p99h4\""
Sep 17 18:38:34.986453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.986453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:34.986453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-node-log\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.986607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-cnibin\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.986607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.986607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-systemd\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.986749 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-system-cni-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.986749 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-socket-dir-parent\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.986749 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-netns\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.986749 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-registration-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-device-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vch\" (UniqueName: \"kubernetes.io/projected/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-kube-api-access-r9vch\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-ovnkube-script-lib\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-kubelet\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:38:34.986934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-var-lib-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvwf\" (UniqueName: \"kubernetes.io/projected/7424f0ba-a16a-4291-936b-391bcb6fdb74-kube-api-access-6xvwf\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.986992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519205-5d21-460c-89ed-53124a87b5fe-hosts-file\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519205-5d21-460c-89ed-53124a87b5fe-tmp-dir\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-kubelet-config\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-run-netns\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-etc-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstcq\" (UniqueName: \"kubernetes.io/projected/4a439def-a26b-458c-952b-86efaf3a282c-kube-api-access-rstcq\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-cnibin\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-cni-multus\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987349    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-k8s-cni-cncf-io\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-cni-bin\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-daemon-config\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-hostroot\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-multus-certs\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6n8\" (UniqueName: \"kubernetes.io/projected/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-kube-api-access-7h6n8\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-system-cni-dir\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-log-socket\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41f96877-8462-4f9c-a21e-1cefab8b45cd-konnectivity-ca\") pod \"konnectivity-agent-vjx5b\" (UID: \"41f96877-8462-4f9c-a21e-1cefab8b45cd\") " pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-sys-fs\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-cni-bin\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-systemd-units\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.987782 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-cni-netd\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-ovn\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-env-overrides\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a439def-a26b-458c-952b-86efaf3a282c-ovn-node-metrics-cert\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccc26168-6aad-4eb1-a6df-db17738b8717-cni-binary-copy\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-kubelet\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-ovnkube-config\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4cj\" (UniqueName: \"kubernetes.io/projected/be519205-5d21-460c-89ed-53124a87b5fe-kube-api-access-8j4cj\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.987992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-socket-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988032    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-slash\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-cni-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41f96877-8462-4f9c-a21e-1cefab8b45cd-agent-certs\") pod \"konnectivity-agent-vjx5b\" (UID: \"41f96877-8462-4f9c-a21e-1cefab8b45cd\") " pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:34.988576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-dbus\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:34.989276 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-etc-selinux\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:34.989276 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-conf-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.989276 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-etc-kubernetes\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.989276 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.988272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lzz\" (UniqueName: \"kubernetes.io/projected/ccc26168-6aad-4eb1-a6df-db17738b8717-kube-api-access-v2lzz\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:34.991314 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.991292    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:38:34.991572 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.991520    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.991671 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.991651    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:38:34.991765 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.991702    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5lslv\""
Sep 17 18:38:34.991899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.991882    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4zpkd\""
Sep 17 18:38:34.991956 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.991930    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:38:34.992146 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:34.992127    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:38:35.000371 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.000348    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:38:35.061977 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.061931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal" event={"ID":"8f3ea4320296e328c2dbf6e5b180d445","Type":"ContainerStarted","Data":"192564d9be00684cb54bc65ffa377a87aae2910150c906df3af7aaa704e22a6f"}
Sep 17 18:38:35.063568 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.063542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal" event={"ID":"02b9657d18b7f38a78ec6983e42d11dc","Type":"ContainerStarted","Data":"a978b4ea2fc2f66df9233140679d6cf8a3af51087f3f7608b524b980b5501ff3"}
Sep 17 18:38:35.077830 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.077768    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:38:35.088660 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-cnibin\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.088760 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.088760 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-modprobe-d\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.088760 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-systemd\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.088760 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-system-cni-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.088952 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-socket-dir-parent\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.088952 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-netns\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.088952 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088807    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-systemd\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.088952 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-system-cni-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.088952 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.088814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.089311 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-registration-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.089385 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-device-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.089385 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vch\" (UniqueName: \"kubernetes.io/projected/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-kube-api-access-r9vch\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.089508 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.089508 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.089508 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-device-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.089662 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.089662 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-cnibin\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.089662 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.089662 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-netns\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.089866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-ovnkube-script-lib\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.089866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-socket-dir-parent\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.089866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-kubelet\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.089866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-systemd\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.089866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-var-lib-kubelet\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.089866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jcc\" (UniqueName: \"kubernetes.io/projected/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-kube-api-access-m9jcc\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-var-lib-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-kubelet\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvwf\" (UniqueName: \"kubernetes.io/projected/7424f0ba-a16a-4291-936b-391bcb6fdb74-kube-api-access-6xvwf\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519205-5d21-460c-89ed-53124a87b5fe-hosts-file\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519205-5d21-460c-89ed-53124a87b5fe-tmp-dir\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-kubelet-config\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.089998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-ovnkube-script-lib\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-run-netns\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-etc-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-registration-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstcq\" (UniqueName: \"kubernetes.io/projected/4a439def-a26b-458c-952b-86efaf3a282c-kube-api-access-rstcq\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-cnibin\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-var-lib-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-kubelet-config\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-etc-openvswitch\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090187 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519205-5d21-460c-89ed-53124a87b5fe-hosts-file\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-cni-multus\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-cni-multus\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-run-netns\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-lib-modules\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.090298    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.090369    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:38:35.590336118 +0000 UTC m=+4.876648634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519205-5d21-460c-89ed-53124a87b5fe-tmp-dir\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b01038-bf4c-4648-bf2c-b5a073762284-host\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16b01038-bf4c-4648-bf2c-b5a073762284-serviceca\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-k8s-cni-cncf-io\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.090586    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-cni-bin\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.090637    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:35.590620382 +0000 UTC m=+4.876932880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:35.090994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-var-lib-cni-bin\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-cnibin\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-daemon-config\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-k8s-cni-cncf-io\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysctl-conf\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090713    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828d3595-96af-4fae-9952-f77f3e34a2be-etc-tuned\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828d3595-96af-4fae-9952-f77f3e34a2be-tmp\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-hostroot\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-multus-certs\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-hostroot\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6n8\" (UniqueName: \"kubernetes.io/projected/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-kube-api-access-7h6n8\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-system-cni-dir\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-host-run-multus-certs\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-log-socket\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41f96877-8462-4f9c-a21e-1cefab8b45cd-konnectivity-ca\") pod \"konnectivity-agent-vjx5b\" (UID: \"41f96877-8462-4f9c-a21e-1cefab8b45cd\") " pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-sys-fs\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-system-cni-dir\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.091812 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-host\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-cni-bin\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.090985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-run\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-iptables-alerter-script\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-systemd-units\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-log-socket\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-cni-netd\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-sys\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-sys-fs\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-ovn\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.091937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-env-overrides\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a439def-a26b-458c-952b-86efaf3a282c-ovn-node-metrics-cert\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccc26168-6aad-4eb1-a6df-db17738b8717-cni-binary-copy\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.092658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqwl\" (UniqueName: \"kubernetes.io/projected/16b01038-bf4c-4648-bf2c-b5a073762284-kube-api-access-vsqwl\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-host-slash\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-kubelet\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-ovnkube-config\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4cj\" (UniqueName: \"kubernetes.io/projected/be519205-5d21-460c-89ed-53124a87b5fe-kube-api-access-8j4cj\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-socket-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysconfig\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41f96877-8462-4f9c-a21e-1cefab8b45cd-konnectivity-ca\") pod \"konnectivity-agent-vjx5b\" (UID: \"41f96877-8462-4f9c-a21e-1cefab8b45cd\") " pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-slash\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-cni-bin\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-slash\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-cni-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41f96877-8462-4f9c-a21e-1cefab8b45cd-agent-certs\") pod \"konnectivity-agent-vjx5b\" (UID: \"41f96877-8462-4f9c-a21e-1cefab8b45cd\") " pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-dbus\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-etc-selinux\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.092940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztdh\" (UniqueName: \"kubernetes.io/projected/828d3595-96af-4fae-9952-f77f3e34a2be-kube-api-access-8ztdh\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.093467 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-conf-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-etc-kubernetes\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lzz\" (UniqueName: \"kubernetes.io/projected/ccc26168-6aad-4eb1-a6df-db17738b8717-kube-api-access-v2lzz\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-node-log\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-kubernetes\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysctl-d\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-cni-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-ovnkube-config\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.093901    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccc26168-6aad-4eb1-a6df-db17738b8717-cni-binary-copy\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-systemd-units\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.094313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-cni-netd\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.094873 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-etc-kubernetes\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.094873 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a439def-a26b-458c-952b-86efaf3a282c-env-overrides\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.094972 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7424f0ba-a16a-4291-936b-391bcb6fdb74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.094972 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.094961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-dbus\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:35.095081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.095064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-etc-selinux\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.095138 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.095120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-conf-dir\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.096555 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.096531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7424f0ba-a16a-4291-936b-391bcb6fdb74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.096895 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.096876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.096981 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.096922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-host-kubelet\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.097109 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.097086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-node-log\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.097342 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.097320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-socket-dir\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.097414 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.097327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a439def-a26b-458c-952b-86efaf3a282c-run-ovn\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.097613 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.097594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccc26168-6aad-4eb1-a6df-db17738b8717-multus-daemon-config\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.099177 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.099110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a439def-a26b-458c-952b-86efaf3a282c-ovn-node-metrics-cert\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.099417 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.099396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vch\" (UniqueName: \"kubernetes.io/projected/d701e8fe-aac2-47cc-ab9c-c9a0c017e009-kube-api-access-r9vch\") pod \"aws-ebs-csi-driver-node-mntl9\" (UID: \"d701e8fe-aac2-47cc-ab9c-c9a0c017e009\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.103674 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.103041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41f96877-8462-4f9c-a21e-1cefab8b45cd-agent-certs\") pod \"konnectivity-agent-vjx5b\" (UID: \"41f96877-8462-4f9c-a21e-1cefab8b45cd\") " pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:35.103674 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.103585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6n8\" (UniqueName: \"kubernetes.io/projected/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-kube-api-access-7h6n8\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:35.104659 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.104284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvwf\" (UniqueName: \"kubernetes.io/projected/7424f0ba-a16a-4291-936b-391bcb6fdb74-kube-api-access-6xvwf\") pod \"multus-additional-cni-plugins-2jnzg\" (UID: \"7424f0ba-a16a-4291-936b-391bcb6fdb74\") " pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.105720 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.105069    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:35.105720 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.105094    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:35.105720 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.105124    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:35.105720 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.105197    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:35.605180148 +0000 UTC m=+4.891492659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:35.106381 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.106332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstcq\" (UniqueName: \"kubernetes.io/projected/4a439def-a26b-458c-952b-86efaf3a282c-kube-api-access-rstcq\") pod \"ovnkube-node-2lf85\" (UID: \"4a439def-a26b-458c-952b-86efaf3a282c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.106664 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.106644    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vstds"
Sep 17 18:38:35.108029 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.108003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4cj\" (UniqueName: \"kubernetes.io/projected/be519205-5d21-460c-89ed-53124a87b5fe-kube-api-access-8j4cj\") pod \"node-resolver-99thn\" (UID: \"be519205-5d21-460c-89ed-53124a87b5fe\") " pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.109224 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.109185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lzz\" (UniqueName: \"kubernetes.io/projected/ccc26168-6aad-4eb1-a6df-db17738b8717-kube-api-access-v2lzz\") pod \"multus-qs64h\" (UID: \"ccc26168-6aad-4eb1-a6df-db17738b8717\") " pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.115359 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.115340    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vstds"
Sep 17 18:38:35.193707 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztdh\" (UniqueName: \"kubernetes.io/projected/828d3595-96af-4fae-9952-f77f3e34a2be-kube-api-access-8ztdh\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.193857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-kubernetes\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.193857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysctl-d\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.193857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-modprobe-d\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.193857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-systemd\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.193857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-var-lib-kubelet\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.193857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jcc\" (UniqueName: \"kubernetes.io/projected/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-kube-api-access-m9jcc\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-lib-modules\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b01038-bf4c-4648-bf2c-b5a073762284-host\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16b01038-bf4c-4648-bf2c-b5a073762284-serviceca\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.193984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysctl-conf\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828d3595-96af-4fae-9952-f77f3e34a2be-etc-tuned\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828d3595-96af-4fae-9952-f77f3e34a2be-tmp\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-host\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-run\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-iptables-alerter-script\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-var-lib-kubelet\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194126 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-sys\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqwl\" (UniqueName: \"kubernetes.io/projected/16b01038-bf4c-4648-bf2c-b5a073762284-kube-api-access-vsqwl\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-host-slash\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysconfig\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-kubernetes\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysctl-d\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-modprobe-d\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-systemd\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-lib-modules\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.194950 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.194638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b01038-bf4c-4648-bf2c-b5a073762284-host\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.195350 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16b01038-bf4c-4648-bf2c-b5a073762284-serviceca\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.195350 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-host-slash\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.195521 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-sys\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.195571 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysctl-conf\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.195619 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-host\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.195670 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-etc-sysconfig\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.195717 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-iptables-alerter-script\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.195717 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.195687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828d3595-96af-4fae-9952-f77f3e34a2be-run\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.197645 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.197605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828d3595-96af-4fae-9952-f77f3e34a2be-tmp\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.199043 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.198988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828d3595-96af-4fae-9952-f77f3e34a2be-etc-tuned\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.203490 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.203428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztdh\" (UniqueName: \"kubernetes.io/projected/828d3595-96af-4fae-9952-f77f3e34a2be-kube-api-access-8ztdh\") pod \"tuned-tzs2x\" (UID: \"828d3595-96af-4fae-9952-f77f3e34a2be\") " pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.204073 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.204053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jcc\" (UniqueName: \"kubernetes.io/projected/ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d-kube-api-access-m9jcc\") pod \"iptables-alerter-g4d9g\" (UID: \"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d\") " pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.205768 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.205737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqwl\" (UniqueName: \"kubernetes.io/projected/16b01038-bf4c-4648-bf2c-b5a073762284-kube-api-access-vsqwl\") pod \"node-ca-f8k74\" (UID: \"16b01038-bf4c-4648-bf2c-b5a073762284\") " pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.213171 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.213152    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:35.271815 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.271791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:35.286017 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.285985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-99thn"
Sep 17 18:38:35.297899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.297876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jnzg"
Sep 17 18:38:35.304678 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.304659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:38:35.310639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.310622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qs64h"
Sep 17 18:38:35.318239 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.318219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9"
Sep 17 18:38:35.326833 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.326813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4d9g"
Sep 17 18:38:35.334475 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.334454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tzs2x"
Sep 17 18:38:35.341092 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.341071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8k74"
Sep 17 18:38:35.597597 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.597524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:35.597597 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.597570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:35.597803 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.597690    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:35.597803 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.597759    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:36.597734255 +0000 UTC m=+5.884046800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:35.597803 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.597690    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:35.597965 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.597845    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:38:36.597824156 +0000 UTC m=+5.884136667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:35.698029 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:35.697990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:35.698167 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.698141    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:35.698167 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.698159    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:35.698167 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.698168    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:35.698300 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:35.698214    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:36.698199825 +0000 UTC m=+5.984512321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:36.056821 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:36.056788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:36.057309 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.056882    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:36.120399 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:36.120365    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:33:35 +0000 UTC" deadline="2028-02-16 08:08:25.315319997 +0000 UTC"
Sep 17 18:38:36.120399 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:36.120393    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12397h29m49.194930374s"
Sep 17 18:38:36.214362 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.214339    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828d3595_96af_4fae_9952_f77f3e34a2be.slice/crio-bc356de3677f3174180d8b41c08a41f46e275f0f088a228cb12d17b514fca8b9 WatchSource:0}: Error finding container bc356de3677f3174180d8b41c08a41f46e275f0f088a228cb12d17b514fca8b9: Status 404 returned error can't find the container with id bc356de3677f3174180d8b41c08a41f46e275f0f088a228cb12d17b514fca8b9
Sep 17 18:38:36.215639 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.215597    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc26168_6aad_4eb1_a6df_db17738b8717.slice/crio-bc8d745374564d272817981f1b64e61df2189d053cb854bc98b0a8091f040561 WatchSource:0}: Error finding container bc8d745374564d272817981f1b64e61df2189d053cb854bc98b0a8091f040561: Status 404 returned error can't find the container with id bc8d745374564d272817981f1b64e61df2189d053cb854bc98b0a8091f040561
Sep 17 18:38:36.216591 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.216565    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a439def_a26b_458c_952b_86efaf3a282c.slice/crio-349274b66d333f1c965754c884c98526409101c02f5489e7c0402e0892162406 WatchSource:0}: Error finding container 349274b66d333f1c965754c884c98526409101c02f5489e7c0402e0892162406: Status 404 returned error can't find the container with id 349274b66d333f1c965754c884c98526409101c02f5489e7c0402e0892162406
Sep 17 18:38:36.218669 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.218232    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd701e8fe_aac2_47cc_ab9c_c9a0c017e009.slice/crio-4531fabda9085f083e5bd09f4c7b3dd6c1bccceb89ad67f5b1fdaa00bbc4c3f2 WatchSource:0}: Error finding container 4531fabda9085f083e5bd09f4c7b3dd6c1bccceb89ad67f5b1fdaa00bbc4c3f2: Status 404 returned error can't find the container with id 4531fabda9085f083e5bd09f4c7b3dd6c1bccceb89ad67f5b1fdaa00bbc4c3f2
Sep 17 18:38:36.240220 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.240194    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7424f0ba_a16a_4291_936b_391bcb6fdb74.slice/crio-11b0975c839e8ce5041e1e5d1c625058b9c2ccd8eec5de55853f020ab2f90189 WatchSource:0}: Error finding container 11b0975c839e8ce5041e1e5d1c625058b9c2ccd8eec5de55853f020ab2f90189: Status 404 returned error can't find the container with id 11b0975c839e8ce5041e1e5d1c625058b9c2ccd8eec5de55853f020ab2f90189
Sep 17 18:38:36.241309 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.241218    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0ec7bd_4def_4f2c_8bc8_6a0e6371a42d.slice/crio-33e65902311b7f514e24d6cc4fcc6b36b00163999b0a245a167a8563524145ff WatchSource:0}: Error finding container 33e65902311b7f514e24d6cc4fcc6b36b00163999b0a245a167a8563524145ff: Status 404 returned error can't find the container with id 33e65902311b7f514e24d6cc4fcc6b36b00163999b0a245a167a8563524145ff
Sep 17 18:38:36.242593 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.242120    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f96877_8462_4f9c_a21e_1cefab8b45cd.slice/crio-a64b64b864471aa1a52a4f0a7bfdbd65b2f3bc754554d7dc5cad81fbc7b5393b WatchSource:0}: Error finding container a64b64b864471aa1a52a4f0a7bfdbd65b2f3bc754554d7dc5cad81fbc7b5393b: Status 404 returned error can't find the container with id a64b64b864471aa1a52a4f0a7bfdbd65b2f3bc754554d7dc5cad81fbc7b5393b
Sep 17 18:38:36.243627 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.243161    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe519205_5d21_460c_89ed_53124a87b5fe.slice/crio-5baee75ea58ccd47adf73aee06bfacf20d473980b76705f5c10c2f2716889462 WatchSource:0}: Error finding container 5baee75ea58ccd47adf73aee06bfacf20d473980b76705f5c10c2f2716889462: Status 404 returned error can't find the container with id 5baee75ea58ccd47adf73aee06bfacf20d473980b76705f5c10c2f2716889462
Sep 17 18:38:36.244215 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:38:36.244054    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b01038_bf4c_4648_bf2c_b5a073762284.slice/crio-b4a6c3d71f5c2600cd54c210e8f944f48ad4cee1d4e80eb66aec637162dc1081 WatchSource:0}: Error finding container b4a6c3d71f5c2600cd54c210e8f944f48ad4cee1d4e80eb66aec637162dc1081: Status 404 returned error can't find the container with id b4a6c3d71f5c2600cd54c210e8f944f48ad4cee1d4e80eb66aec637162dc1081
Sep 17 18:38:36.605141 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:36.604953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:36.605141 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.605106    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:36.605141 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:36.605122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:36.605311 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.605166    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:38:38.605140644 +0000 UTC m=+7.891453139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:36.605311 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.605200    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:36.605311 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.605243    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:38.605230873 +0000 UTC m=+7.891543367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:36.706464 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:36.706417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:36.706633 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.706610    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:36.706721 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.706635    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:36.706721 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.706649    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:36.706721 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:36.706708    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:38.70669007 +0000 UTC m=+7.993002585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:37.057578 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.057550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:37.058255 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.057561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:37.058255 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:37.057663    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:37.058255 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:37.057771    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:37.072261 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.072230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal" event={"ID":"8f3ea4320296e328c2dbf6e5b180d445","Type":"ContainerStarted","Data":"253c60c92b136b429d31ea591dbffb82ca2c5446a33ca48c428160847ad06acc"}
Sep 17 18:38:37.079752 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.079725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8k74" event={"ID":"16b01038-bf4c-4648-bf2c-b5a073762284","Type":"ContainerStarted","Data":"b4a6c3d71f5c2600cd54c210e8f944f48ad4cee1d4e80eb66aec637162dc1081"}
Sep 17 18:38:37.087988 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.087963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4d9g" event={"ID":"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d","Type":"ContainerStarted","Data":"33e65902311b7f514e24d6cc4fcc6b36b00163999b0a245a167a8563524145ff"}
Sep 17 18:38:37.093240 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.093210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerStarted","Data":"11b0975c839e8ce5041e1e5d1c625058b9c2ccd8eec5de55853f020ab2f90189"}
Sep 17 18:38:37.098862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.098832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qs64h" event={"ID":"ccc26168-6aad-4eb1-a6df-db17738b8717","Type":"ContainerStarted","Data":"bc8d745374564d272817981f1b64e61df2189d053cb854bc98b0a8091f040561"}
Sep 17 18:38:37.110396 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.110198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9" event={"ID":"d701e8fe-aac2-47cc-ab9c-c9a0c017e009","Type":"ContainerStarted","Data":"4531fabda9085f083e5bd09f4c7b3dd6c1bccceb89ad67f5b1fdaa00bbc4c3f2"}
Sep 17 18:38:37.118425 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.118328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"349274b66d333f1c965754c884c98526409101c02f5489e7c0402e0892162406"}
Sep 17 18:38:37.120795 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.120742    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:33:35 +0000 UTC" deadline="2028-03-19 07:35:49.264291041 +0000 UTC"
Sep 17 18:38:37.120795 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.120774    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13164h57m12.143520374s"
Sep 17 18:38:37.126362 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.126285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tzs2x" event={"ID":"828d3595-96af-4fae-9952-f77f3e34a2be","Type":"ContainerStarted","Data":"bc356de3677f3174180d8b41c08a41f46e275f0f088a228cb12d17b514fca8b9"}
Sep 17 18:38:37.135603 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.135479    2565 generic.go:356] "Generic (PLEG): container finished" podID="02b9657d18b7f38a78ec6983e42d11dc" containerID="a8554ccfc152bdc8f5d95104714fe64e952c02593a2beb641771be45a5f93826" exitCode=0
Sep 17 18:38:37.135603 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.135560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal" event={"ID":"02b9657d18b7f38a78ec6983e42d11dc","Type":"ContainerDied","Data":"a8554ccfc152bdc8f5d95104714fe64e952c02593a2beb641771be45a5f93826"}
Sep 17 18:38:37.148874 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.148847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-99thn" event={"ID":"be519205-5d21-460c-89ed-53124a87b5fe","Type":"ContainerStarted","Data":"5baee75ea58ccd47adf73aee06bfacf20d473980b76705f5c10c2f2716889462"}
Sep 17 18:38:37.153978 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.153949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vjx5b" event={"ID":"41f96877-8462-4f9c-a21e-1cefab8b45cd","Type":"ContainerStarted","Data":"a64b64b864471aa1a52a4f0a7bfdbd65b2f3bc754554d7dc5cad81fbc7b5393b"}
Sep 17 18:38:37.169448 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:37.169393    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-95.ec2.internal" podStartSLOduration=3.169375619 podStartE2EDuration="3.169375619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:37.090202194 +0000 UTC m=+6.376514706" watchObservedRunningTime="2026-09-17 18:38:37.169375619 +0000 UTC m=+6.455688129"
Sep 17 18:38:38.057284 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:38.057257    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:38.057474 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.057375    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:38.171532 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:38.171492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal" event={"ID":"02b9657d18b7f38a78ec6983e42d11dc","Type":"ContainerStarted","Data":"349fc96895fbee16130ed260c29ae45c5937acde8bf44e34f79ea618c2feba98"}
Sep 17 18:38:38.625103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:38.625067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:38.625269 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:38.625120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:38.625394 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.625374    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:38.625486 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.625464    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:38:42.62542971 +0000 UTC m=+11.911742225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:38.625874 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.625854    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:38.625977 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.625905    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:42.625892599 +0000 UTC m=+11.912205096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:38.725461 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:38.725406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:38.725667 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.725643    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:38.725738 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.725676    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:38.725738 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.725689    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:38.725858 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:38.725746    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:42.725726941 +0000 UTC m=+12.012039454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:39.057566 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:39.057537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:39.057728 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:39.057670    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:39.058216 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:39.058196    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:39.058330 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:39.058311    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:40.057188 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:40.057156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:40.057671 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:40.057282    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:41.057281 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:41.057250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:41.057758 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:41.057372    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:41.057758 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:41.057506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:41.057758 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:41.057624    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:42.056866 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:42.056835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:42.057031 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.056963    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:42.659939 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:42.659901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:42.660385 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:42.659954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:42.660385 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.660077    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:42.660385 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.660121    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:42.660385 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.660150    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:38:50.660131685 +0000 UTC m=+19.946444192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:42.660385 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.660175    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:50.660158778 +0000 UTC m=+19.946471288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:42.761285 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:42.761251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:42.761467 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.761450    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:42.761535 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.761476    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:42.761535 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.761489    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:42.761649 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:42.761543    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:50.761526738 +0000 UTC m=+20.047839253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:43.058487 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:43.058457    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:43.058487 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:43.058488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:43.058743 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:43.058583    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:43.058743 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:43.058727    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:44.057033 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:44.056755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:44.057033 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:44.056879    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:45.056981 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:45.056949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:45.057159 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:45.056998    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:45.057159 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:45.057080    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:45.057602 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:45.057238    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:46.057031 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:46.056999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:46.057240 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:46.057116    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:47.057547 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:47.057516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:47.057975 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:47.057523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:47.057975 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:47.057655    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:47.057975 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:47.057748    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:48.056671 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:48.056641    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:48.056831 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:48.056752    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:49.056753 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:49.056706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:49.057188 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:49.056720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:49.057188 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:49.056831    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:49.057188 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:49.056915    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:50.057432 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:50.057401    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:50.057845 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.057519    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:50.718422 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:50.718386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:50.718601 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:50.718452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:50.718601 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.718587    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:50.718726 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.718564    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:50.718726 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.718640    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:06.718625861 +0000 UTC m=+36.004938356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:50.718726 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.718681    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:06.718663553 +0000 UTC m=+36.004976051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:50.818921 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:50.818892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:50.819077 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.819046    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:50.819077 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.819063    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:50.819077 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.819073    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:50.819221 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:50.819124    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:06.819106002 +0000 UTC m=+36.105418511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:51.057537 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:51.057464    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:51.057537 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:51.057489    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:51.058037 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:51.057583    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:51.058037 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:51.057715    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:52.057475 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:52.057425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:52.057622 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:52.057572    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:53.057269 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:53.057239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:53.057470 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:53.057330    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:53.057470 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:53.057406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:53.057583 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:53.057502    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:54.057292 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.057145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:54.057882 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:54.057367    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:54.207838 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.207811    2565 generic.go:356] "Generic (PLEG): container finished" podID="7424f0ba-a16a-4291-936b-391bcb6fdb74" containerID="bb37ffd320ba826e08341f80c59286e4e4b2c6b23a8d25743f2c700741f370f9" exitCode=0
Sep 17 18:38:54.207942 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.207872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerDied","Data":"bb37ffd320ba826e08341f80c59286e4e4b2c6b23a8d25743f2c700741f370f9"}
Sep 17 18:38:54.209619 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.209559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qs64h" event={"ID":"ccc26168-6aad-4eb1-a6df-db17738b8717","Type":"ContainerStarted","Data":"c1d4b6b756b924b7c00596e5f47fa29b584a00da69c70ea4d6dc7ee5cdfcffe0"}
Sep 17 18:38:54.211110 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.211087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9" event={"ID":"d701e8fe-aac2-47cc-ab9c-c9a0c017e009","Type":"ContainerStarted","Data":"cd9973f082681829f75559474fe3483bacdfa7383586f54494052ac78c458bac"}
Sep 17 18:38:54.214620 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.214595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"4b9d90481a804a8a8340c79761500e1390033e61bc7d96ee394876a6663d802c"}
Sep 17 18:38:54.214690 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.214670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"0d8e9cd26db6d1469cba22d1edbadeb83e61ad91354f3f006c7baf3fa753f114"}
Sep 17 18:38:54.214738 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.214723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"1f202318116ef511d18410cd3b089c402926dcfdff3b6f07a0f4b8b26301772d"}
Sep 17 18:38:54.214789 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.214776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"e475300705739c3b0ae52dce430288e22dc1a557243925142d34000a44fd3ed4"}
Sep 17 18:38:54.214841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.214827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"598289ec6e59a976c21d962c44089d38bef57c2be22bf57ed515908bb61c7820"}
Sep 17 18:38:54.214886 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.214864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"8d3adacbad1eb3772a7373ba30a3ef30eb7daa515ff1d674ef7713c90f19e90a"}
Sep 17 18:38:54.216073 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.216052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tzs2x" event={"ID":"828d3595-96af-4fae-9952-f77f3e34a2be","Type":"ContainerStarted","Data":"6c811810cef921e57d2ef9b85068a99e826ae68f4bb7902e3e5efd678e12f04c"}
Sep 17 18:38:54.217508 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.217486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-99thn" event={"ID":"be519205-5d21-460c-89ed-53124a87b5fe","Type":"ContainerStarted","Data":"a7fc14056b7ae6edb6149f89fa9cdb167eb6296b69e8b5b4f05a005f7dc9a3da"}
Sep 17 18:38:54.218837 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.218818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vjx5b" event={"ID":"41f96877-8462-4f9c-a21e-1cefab8b45cd","Type":"ContainerStarted","Data":"c813432b4fbd893a8ce5e0ddf887b5839d8bfeea13e370325b0fd1713227c495"}
Sep 17 18:38:54.220197 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.220176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8k74" event={"ID":"16b01038-bf4c-4648-bf2c-b5a073762284","Type":"ContainerStarted","Data":"10c31d5ac20776264bdf72f421d9eaabc537594597a7f1c6dede74dfeaf3eb3a"}
Sep 17 18:38:54.229729 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.229697    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-95.ec2.internal" podStartSLOduration=20.22968688 podStartE2EDuration="20.22968688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:38.189301776 +0000 UTC m=+7.475614294" watchObservedRunningTime="2026-09-17 18:38:54.22968688 +0000 UTC m=+23.515999396"
Sep 17 18:38:54.258464 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.258412    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tzs2x" podStartSLOduration=4.110802248 podStartE2EDuration="21.258396012s" totalImagesPullingTime="17.147593764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:54.244220205 +0000 UTC m=+23.530532722" watchObservedRunningTime="2026-09-17 18:38:54.258396012 +0000 UTC m=+23.544708529"
Sep 17 18:38:54.258536 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.258518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f8k74" podStartSLOduration=4.143036614 podStartE2EDuration="21.258514932s" totalImagesPullingTime="17.115478318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:54.258362629 +0000 UTC m=+23.544675146" watchObservedRunningTime="2026-09-17 18:38:54.258514932 +0000 UTC m=+23.544827449"
Sep 17 18:38:54.276853 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.276821    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vjx5b" podStartSLOduration=8.18928962 podStartE2EDuration="21.276810627s" totalImagesPullingTime="13.087521007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:54.276448388 +0000 UTC m=+23.562760906" watchObservedRunningTime="2026-09-17 18:38:54.276810627 +0000 UTC m=+23.563123144"
Sep 17 18:38:54.301589 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.301528    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-99thn" podStartSLOduration=8.213930926 podStartE2EDuration="21.301517075s" totalImagesPullingTime="13.087586149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:54.301088366 +0000 UTC m=+23.587400886" watchObservedRunningTime="2026-09-17 18:38:54.301517075 +0000 UTC m=+23.587829591"
Sep 17 18:38:54.328314 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.328278    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qs64h" podStartSLOduration=4.122426112 podStartE2EDuration="21.328264584s" totalImagesPullingTime="17.205838472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:54.32800125 +0000 UTC m=+23.614313769" watchObservedRunningTime="2026-09-17 18:38:54.328264584 +0000 UTC m=+23.614577104"
Sep 17 18:38:54.784052 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:54.783894    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:38:55.056982 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.056957    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:55.057125 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.056957    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:55.057125 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:55.057061    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:55.057210 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:55.057157    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:55.065664 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.065596    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:38:54.784048651Z","UUID":"45479849-3f30-4476-9f58-fecc82dd5f27","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:38:55.066864 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.066845    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:38:55.066972 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.066878    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:38:55.223278 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.223248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4d9g" event={"ID":"ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d","Type":"ContainerStarted","Data":"b6d20355e435a2757b5c86e451fbeb66d276b7ed16e996d13efa8d78f7ea3d5a"}
Sep 17 18:38:55.227859 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.227830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9" event={"ID":"d701e8fe-aac2-47cc-ab9c-c9a0c017e009","Type":"ContainerStarted","Data":"254d05fb356984440cdbd77f8d5be63f05918aaa307a51af4a3b9a399b46e046"}
Sep 17 18:38:55.239223 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:55.239184    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g4d9g" podStartSLOduration=5.12190482 podStartE2EDuration="22.23917253s" totalImagesPullingTime="17.11726771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:55.238784963 +0000 UTC m=+24.525097479" watchObservedRunningTime="2026-09-17 18:38:55.23917253 +0000 UTC m=+24.525485047"
Sep 17 18:38:56.013279 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.013249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:56.013950 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.013931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:56.057454 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.057418    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:56.057559 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:56.057514    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:56.231961 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.231933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9" event={"ID":"d701e8fe-aac2-47cc-ab9c-c9a0c017e009","Type":"ContainerStarted","Data":"95e3982ddc3714e0f26a622ad10c84ab017cfd437c4f412bd4b0cc68148139d5"}
Sep 17 18:38:56.235109 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.235086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"76a4671e7a45bddada20320465d1be1871f99072ca7a90bff739a8448c820c52"}
Sep 17 18:38:56.235377 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.235353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:56.235774 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.235760    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vjx5b"
Sep 17 18:38:56.250182 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:56.250148    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mntl9" podStartSLOduration=3.978114808 podStartE2EDuration="23.250136701s" totalImagesPullingTime="19.272021893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:56.249842436 +0000 UTC m=+25.536154952" watchObservedRunningTime="2026-09-17 18:38:56.250136701 +0000 UTC m=+25.536449216"
Sep 17 18:38:57.057031 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:57.057001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:57.057184 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:57.057098    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:57.057184 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:57.057152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:57.057253 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:57.057237    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:38:58.057560 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:58.057528    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:38:58.057912 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:58.057647    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:38:59.060197 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:59.060163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:38:59.060658 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:59.060276    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:38:59.060658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:38:59.060327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:38:59.060658 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:38:59.060430    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:39:00.057106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.056927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:00.057204 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:00.057186    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:39:00.245402 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.245375    2565 generic.go:356] "Generic (PLEG): container finished" podID="7424f0ba-a16a-4291-936b-391bcb6fdb74" containerID="7036e0a8956e58a67dd0812f3964b259dd28e2a80b9b31f5c09d978de8ba3bc9" exitCode=0
Sep 17 18:39:00.245997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.245465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerDied","Data":"7036e0a8956e58a67dd0812f3964b259dd28e2a80b9b31f5c09d978de8ba3bc9"}
Sep 17 18:39:00.249231 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.249211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" event={"ID":"4a439def-a26b-458c-952b-86efaf3a282c","Type":"ContainerStarted","Data":"b852793fd054a04d7f51bf092bf1fdb4114b7c0297af130bf8098cd07ef925ec"}
Sep 17 18:39:00.249480 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.249465    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:39:00.249558 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.249514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:39:00.263462 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.263429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:39:00.300570 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:00.300533    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85" podStartSLOduration=10.125522958 podStartE2EDuration="27.300521179s" totalImagesPullingTime="17.174998221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:00.300380958 +0000 UTC m=+29.586693472" watchObservedRunningTime="2026-09-17 18:39:00.300521179 +0000 UTC m=+29.586833695"
Sep 17 18:39:01.057571 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.057543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:01.057787 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:01.057670    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:39:01.058030 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.058008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:01.058145 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:01.058113    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:39:01.253491 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.253411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:39:01.267592 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.267567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:39:01.427771 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.427379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gvpvp"]
Sep 17 18:39:01.427771 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.427528    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:01.427771 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:01.427627    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:39:01.428124 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.428108    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pzcln"]
Sep 17 18:39:01.428197 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.428183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:01.428278 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:01.428257    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:39:01.428691 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.428669    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhxpv"]
Sep 17 18:39:01.428779 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:01.428768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:01.428856 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:01.428838    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:39:02.257197 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:02.257026    2565 generic.go:356] "Generic (PLEG): container finished" podID="7424f0ba-a16a-4291-936b-391bcb6fdb74" containerID="b217f431025c5bc49940de5d2f8b8d933b6eb9ee7468a963d154ac9948c2f6d7" exitCode=0
Sep 17 18:39:02.257583 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:02.257108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerDied","Data":"b217f431025c5bc49940de5d2f8b8d933b6eb9ee7468a963d154ac9948c2f6d7"}
Sep 17 18:39:03.057919 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:03.057846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:03.058091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:03.057927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:03.058091 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:03.057938    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:39:03.058091 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:03.058047    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:39:03.058091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:03.058075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:03.058274 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:03.058134    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:39:04.263239 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:04.263211    2565 generic.go:356] "Generic (PLEG): container finished" podID="7424f0ba-a16a-4291-936b-391bcb6fdb74" containerID="aea5db614e64c276b4dce95740ecc5d7f993a78725db740638819776c8dc16d3" exitCode=0
Sep 17 18:39:04.263570 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:04.263269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerDied","Data":"aea5db614e64c276b4dce95740ecc5d7f993a78725db740638819776c8dc16d3"}
Sep 17 18:39:05.057549 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:05.057517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:05.057719 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:05.057611    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzcln" podUID="d63ac213-96df-4e51-9fd9-b76edfb86467"
Sep 17 18:39:05.057719 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:05.057639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:05.057719 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:05.057517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:05.057853 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:05.057728    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvpvp" podUID="433bbf08-f013-4ad0-8fd0-f592eba5ac4c"
Sep 17 18:39:05.057853 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:05.057790    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhxpv" podUID="0255f4fa-4df9-439c-8cc2-5b921c8a64a1"
Sep 17 18:39:06.456661 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.456638    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-95.ec2.internal" event="NodeReady"
Sep 17 18:39:06.456990 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.456722    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:39:06.495878 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.495852    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d4f5d4d78-j6smq"]
Sep 17 18:39:06.520713 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.520696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.523105 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.523085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:39:06.523558 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.523538    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g9jcp"]
Sep 17 18:39:06.524068 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.524047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dw5r6\""
Sep 17 18:39:06.524491 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.524287    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:39:06.524860 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.524829    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:39:06.529382 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.529363    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:39:06.548763 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.548743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4f5d4d78-j6smq"]
Sep 17 18:39:06.548907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.548781    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hpkk2"]
Sep 17 18:39:06.548907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.548845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.551272 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.551251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zttzk\""
Sep 17 18:39:06.551632 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.551613    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:39:06.551730 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.551633    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:39:06.567084 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.567065    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g9jcp"]
Sep 17 18:39:06.567193 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.567105    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpkk2"]
Sep 17 18:39:06.567193 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.567173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.569762 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.569735    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:39:06.569870 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.569851    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:39:06.569951 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.569919    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:39:06.570018 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.569950    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jpf78\""
Sep 17 18:39:06.631990 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.631969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-installation-pull-secrets\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632104 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkl6\" (UniqueName: \"kubernetes.io/projected/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-kube-api-access-zlkl6\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.632104 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-image-registry-private-configuration\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632104 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-bound-sa-token\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632104 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-ca-trust-extracted\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-certificates\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-tmp-dir\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.632296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-config-volume\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.632296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-trusted-ca\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpvpx\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-kube-api-access-xpvpx\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.632516 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.632353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.732891 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkl6\" (UniqueName: \"kubernetes.io/projected/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-kube-api-access-zlkl6\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.732891 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-image-registry-private-configuration\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.732891 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-bound-sa-token\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-ca-trust-extracted\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.732993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-certificates\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-tmp-dir\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733038    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-config-volume\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.733103 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733101    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret podName:433bbf08-f013-4ad0-8fd0-f592eba5ac4c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.733081863 +0000 UTC m=+68.019394379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret") pod "global-pull-secret-syncer-gvpvp" (UID: "433bbf08-f013-4ad0-8fd0-f592eba5ac4c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4d4de15-fc5d-48c9-aa48-da635f088ef7-cert\") pod \"ingress-canary-hpkk2\" (UID: \"e4d4de15-fc5d-48c9-aa48-da635f088ef7\") " pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733187    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-trusted-ca\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpvpx\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-kube-api-access-xpvpx\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733250    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.733233966 +0000 UTC m=+68.019546475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-ca-trust-extracted\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-installation-pull-secrets\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8fb\" (UniqueName: \"kubernetes.io/projected/e4d4de15-fc5d-48c9-aa48-da635f088ef7-kube-api-access-4b8fb\") pod \"ingress-canary-hpkk2\" (UID: \"e4d4de15-fc5d-48c9-aa48-da635f088ef7\") " pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733404    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733418    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733490    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:07.233476399 +0000 UTC m=+36.519788913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-tmp-dir\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.733550 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.733517    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:06.734016 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-config-volume\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.734016 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.733914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-certificates\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.734078 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.734068    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:39:07.234056868 +0000 UTC m=+36.520369365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:06.734306 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.734289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-trusted-ca\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.736734 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.736715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-image-registry-private-configuration\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.736734 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.736728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-installation-pull-secrets\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.742929 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.742910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-bound-sa-token\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.743076 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.743059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpvpx\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-kube-api-access-xpvpx\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:06.743283 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.743266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkl6\" (UniqueName: \"kubernetes.io/projected/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-kube-api-access-zlkl6\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:06.834369 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.834344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:06.834509 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.834376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8fb\" (UniqueName: \"kubernetes.io/projected/e4d4de15-fc5d-48c9-aa48-da635f088ef7-kube-api-access-4b8fb\") pod \"ingress-canary-hpkk2\" (UID: \"e4d4de15-fc5d-48c9-aa48-da635f088ef7\") " pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.834509 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.834466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4d4de15-fc5d-48c9-aa48-da635f088ef7-cert\") pod \"ingress-canary-hpkk2\" (UID: \"e4d4de15-fc5d-48c9-aa48-da635f088ef7\") " pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.834509 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.834502    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:06.834665 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.834526    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:06.834665 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.834539    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rrdxm for pod openshift-network-diagnostics/network-check-target-pzcln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:06.834665 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:06.834589    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm podName:d63ac213-96df-4e51-9fd9-b76edfb86467 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.834575936 +0000 UTC m=+68.120888431 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrdxm" (UniqueName: "kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm") pod "network-check-target-pzcln" (UID: "d63ac213-96df-4e51-9fd9-b76edfb86467") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:06.836908 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.836878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4d4de15-fc5d-48c9-aa48-da635f088ef7-cert\") pod \"ingress-canary-hpkk2\" (UID: \"e4d4de15-fc5d-48c9-aa48-da635f088ef7\") " pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.844601 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.844584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8fb\" (UniqueName: \"kubernetes.io/projected/e4d4de15-fc5d-48c9-aa48-da635f088ef7-kube-api-access-4b8fb\") pod \"ingress-canary-hpkk2\" (UID: \"e4d4de15-fc5d-48c9-aa48-da635f088ef7\") " pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:06.876690 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:06.876667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpkk2"
Sep 17 18:39:07.056840 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.056665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:07.056979 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.056665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:07.057033 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.056701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:07.059986 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.059694    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kfmzw\""
Sep 17 18:39:07.059986 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.059780    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:39:07.059986 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.059924    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:39:07.060607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.060582    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:07.060607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.060603    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gk2t2\""
Sep 17 18:39:07.060765 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.060641    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:07.212946 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.212919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpkk2"]
Sep 17 18:39:07.216976 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:07.216949    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d4de15_fc5d_48c9_aa48_da635f088ef7.slice/crio-b2c615eb9c15ffc7774b3bf8bf41c688c8ec8da6479033ec8fe051847f947517 WatchSource:0}: Error finding container b2c615eb9c15ffc7774b3bf8bf41c688c8ec8da6479033ec8fe051847f947517: Status 404 returned error can't find the container with id b2c615eb9c15ffc7774b3bf8bf41c688c8ec8da6479033ec8fe051847f947517
Sep 17 18:39:07.238228 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.238175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:07.238228 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.238211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:07.238355 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:07.238317    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:07.238355 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:07.238334    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:07.238355 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:07.238335    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:07.238484 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:07.238392    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:39:08.23837663 +0000 UTC m=+37.524689125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:07.238484 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:07.238410    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:08.238401813 +0000 UTC m=+37.524714311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:07.270823 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:07.270793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpkk2" event={"ID":"e4d4de15-fc5d-48c9-aa48-da635f088ef7","Type":"ContainerStarted","Data":"b2c615eb9c15ffc7774b3bf8bf41c688c8ec8da6479033ec8fe051847f947517"}
Sep 17 18:39:08.244153 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:08.244119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:08.244153 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:08.244159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:08.244723 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:08.244272    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:08.244723 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:08.244328    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:39:10.244310007 +0000 UTC m=+39.530622525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:08.244723 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:08.244391    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:08.244723 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:08.244413    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:08.244723 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:08.244484    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:10.244467816 +0000 UTC m=+39.530780315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:10.260215 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:10.260187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:10.260799 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:10.260233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:10.260799 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:10.260349    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:10.260799 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:10.260368    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:10.260799 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:10.260425    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:14.260410153 +0000 UTC m=+43.546722651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:10.260799 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:10.260425    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:10.260799 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:10.260492    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:39:14.260477986 +0000 UTC m=+43.546790480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:10.280739 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:10.280715    2565 generic.go:356] "Generic (PLEG): container finished" podID="7424f0ba-a16a-4291-936b-391bcb6fdb74" containerID="c8b9ce2f5fec0cb3fe43de3673648224a8217eec7aafae0406dd7cf509f1ce7a" exitCode=0
Sep 17 18:39:10.280884 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:10.280778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerDied","Data":"c8b9ce2f5fec0cb3fe43de3673648224a8217eec7aafae0406dd7cf509f1ce7a"}
Sep 17 18:39:11.287547 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:11.287374    2565 generic.go:356] "Generic (PLEG): container finished" podID="7424f0ba-a16a-4291-936b-391bcb6fdb74" containerID="3a94c647ee88385613896375018be7b90ccd8a8552dbad6b5e362ceefbb2bd9b" exitCode=0
Sep 17 18:39:11.287547 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:11.287451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerDied","Data":"3a94c647ee88385613896375018be7b90ccd8a8552dbad6b5e362ceefbb2bd9b"}
Sep 17 18:39:12.292780 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:12.292743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" event={"ID":"7424f0ba-a16a-4291-936b-391bcb6fdb74","Type":"ContainerStarted","Data":"c98e93fe7570e9f6d783cc121c47d24ec7020db511ff27c286350ee73f61fb8c"}
Sep 17 18:39:12.294141 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:12.294119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpkk2" event={"ID":"e4d4de15-fc5d-48c9-aa48-da635f088ef7","Type":"ContainerStarted","Data":"8f592a697780c479bd963ff36625dfb25e27dd6b76372dc95a53c9656a4b96b5"}
Sep 17 18:39:12.318218 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:12.318177    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2jnzg" podStartSLOduration=13.192390024 podStartE2EDuration="39.318161688s" totalImagesPullingTime="26.125771664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:12.317201115 +0000 UTC m=+41.603513628" watchObservedRunningTime="2026-09-17 18:39:12.318161688 +0000 UTC m=+41.604474198"
Sep 17 18:39:12.338515 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:12.338481    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hpkk2" podStartSLOduration=1.950354768 podStartE2EDuration="6.338468541s" totalImagesPullingTime="4.388113773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:12.338348215 +0000 UTC m=+41.624660732" watchObservedRunningTime="2026-09-17 18:39:12.338468541 +0000 UTC m=+41.624781087"
Sep 17 18:39:14.292697 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:14.292663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:14.293080 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:14.292706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:14.293080 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:14.292793    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:14.293080 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:14.292806    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:14.293080 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:14.292811    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:14.293080 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:14.292857    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.292845242 +0000 UTC m=+51.579157736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:14.293080 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:14.292891    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.292868098 +0000 UTC m=+51.579180600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.347110 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:22.347066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:22.347565 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:22.347124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:22.347565 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:22.347218    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:22.347565 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:22.347237    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:22.347565 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:22.347274    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.347565 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:22.347286    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.347272428 +0000 UTC m=+67.633584924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:22.347565 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:22.347319    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.347307348 +0000 UTC m=+67.633619843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:25.662688 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.662649    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"]
Sep 17 18:39:25.678007 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.677981    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"]
Sep 17 18:39:25.678129 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.678085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"
Sep 17 18:39:25.680316 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.680299    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r42l2\""
Sep 17 18:39:25.692127 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.692109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"]
Sep 17 18:39:25.692207 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.692137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"]
Sep 17 18:39:25.692207 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.692199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.695905 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.695872    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 18:39:25.696036 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.695885    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:39:25.696939 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.696920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ksb7k\""
Sep 17 18:39:25.697317 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.697188    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:39:25.697317 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.697226    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 18:39:25.759357 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.759333    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"]
Sep 17 18:39:25.780007 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.779990    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"]
Sep 17 18:39:25.780095 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.780065    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"
Sep 17 18:39:25.782544 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.782525    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:25.783468 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.782837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-blkcl\""
Sep 17 18:39:25.783468 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.783061    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:25.868784 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.868758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29d5\" (UniqueName: \"kubernetes.io/projected/dcf0ad40-19d2-44f9-a69d-911b05e056e9-kube-api-access-d29d5\") pod \"network-check-source-7cbc45fddc-km4bg\" (UID: \"dcf0ad40-19d2-44f9-a69d-911b05e056e9\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"
Sep 17 18:39:25.868903 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.868787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsxg\" (UniqueName: \"kubernetes.io/projected/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-kube-api-access-5hsxg\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.868903 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.868806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.868903 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.868828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.969277 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.969251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29d5\" (UniqueName: \"kubernetes.io/projected/dcf0ad40-19d2-44f9-a69d-911b05e056e9-kube-api-access-d29d5\") pod \"network-check-source-7cbc45fddc-km4bg\" (UID: \"dcf0ad40-19d2-44f9-a69d-911b05e056e9\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"
Sep 17 18:39:25.969384 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.969285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsxg\" (UniqueName: \"kubernetes.io/projected/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-kube-api-access-5hsxg\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.969384 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.969303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.969384 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.969332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.969384 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.969356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvsr\" (UniqueName: \"kubernetes.io/projected/61c70057-1bab-4c67-9d06-60c5d910329a-kube-api-access-zwvsr\") pod \"volume-data-source-validator-5c4db8bdf9-rg9g7\" (UID: \"61c70057-1bab-4c67-9d06-60c5d910329a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"
Sep 17 18:39:25.969646 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:25.969459    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:25.969646 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:25.969547    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls podName:9d4f84b2-13fa-465c-bca2-aa7f8027c5f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:26.469524075 +0000 UTC m=+55.755836587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-ddvnb" (UID: "9d4f84b2-13fa-465c-bca2-aa7f8027c5f5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:25.970052 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.970032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.985200 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.985168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsxg\" (UniqueName: \"kubernetes.io/projected/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-kube-api-access-5hsxg\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:25.985306 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.985283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29d5\" (UniqueName: \"kubernetes.io/projected/dcf0ad40-19d2-44f9-a69d-911b05e056e9-kube-api-access-d29d5\") pod \"network-check-source-7cbc45fddc-km4bg\" (UID: \"dcf0ad40-19d2-44f9-a69d-911b05e056e9\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"
Sep 17 18:39:25.988008 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:25.987995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"
Sep 17 18:39:26.070319 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.070286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvsr\" (UniqueName: \"kubernetes.io/projected/61c70057-1bab-4c67-9d06-60c5d910329a-kube-api-access-zwvsr\") pod \"volume-data-source-validator-5c4db8bdf9-rg9g7\" (UID: \"61c70057-1bab-4c67-9d06-60c5d910329a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"
Sep 17 18:39:26.086590 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.086565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvsr\" (UniqueName: \"kubernetes.io/projected/61c70057-1bab-4c67-9d06-60c5d910329a-kube-api-access-zwvsr\") pod \"volume-data-source-validator-5c4db8bdf9-rg9g7\" (UID: \"61c70057-1bab-4c67-9d06-60c5d910329a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"
Sep 17 18:39:26.089172 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.089154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"
Sep 17 18:39:26.270982 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.270930    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg"]
Sep 17 18:39:26.273541 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:26.273519    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf0ad40_19d2_44f9_a69d_911b05e056e9.slice/crio-86f2ca15ee70be2ebae3f6578f272edac665171ed809164d81518d096f1c4fd0 WatchSource:0}: Error finding container 86f2ca15ee70be2ebae3f6578f272edac665171ed809164d81518d096f1c4fd0: Status 404 returned error can't find the container with id 86f2ca15ee70be2ebae3f6578f272edac665171ed809164d81518d096f1c4fd0
Sep 17 18:39:26.325297 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.325274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg" event={"ID":"dcf0ad40-19d2-44f9-a69d-911b05e056e9","Type":"ContainerStarted","Data":"86f2ca15ee70be2ebae3f6578f272edac665171ed809164d81518d096f1c4fd0"}
Sep 17 18:39:26.372024 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.371998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7"]
Sep 17 18:39:26.375180 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:26.375158    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c70057_1bab_4c67_9d06_60c5d910329a.slice/crio-bce0fdca7b6edcc9ece635f43d40ac0b9abfca9e572a1867f2e07af40e213d39 WatchSource:0}: Error finding container bce0fdca7b6edcc9ece635f43d40ac0b9abfca9e572a1867f2e07af40e213d39: Status 404 returned error can't find the container with id bce0fdca7b6edcc9ece635f43d40ac0b9abfca9e572a1867f2e07af40e213d39
Sep 17 18:39:26.474080 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:26.474048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:26.474204 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:26.474186    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:26.474265 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:26.474255    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls podName:9d4f84b2-13fa-465c-bca2-aa7f8027c5f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:27.474236528 +0000 UTC m=+56.760549025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-ddvnb" (UID: "9d4f84b2-13fa-465c-bca2-aa7f8027c5f5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:27.329574 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:27.329535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7" event={"ID":"61c70057-1bab-4c67-9d06-60c5d910329a","Type":"ContainerStarted","Data":"bce0fdca7b6edcc9ece635f43d40ac0b9abfca9e572a1867f2e07af40e213d39"}
Sep 17 18:39:27.480414 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:27.480378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:27.480622 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:27.480548    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:27.480695 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:27.480634    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls podName:9d4f84b2-13fa-465c-bca2-aa7f8027c5f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:29.480614328 +0000 UTC m=+58.766926828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-ddvnb" (UID: "9d4f84b2-13fa-465c-bca2-aa7f8027c5f5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:28.334313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:28.334226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7" event={"ID":"61c70057-1bab-4c67-9d06-60c5d910329a","Type":"ContainerStarted","Data":"88b406a5267e9da7bbf0d42fdab7e46961b4fd0d2ef3439d4935f5f594a54838"}
Sep 17 18:39:28.353284 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:28.353237    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rg9g7" podStartSLOduration=1.7046230169999999 podStartE2EDuration="3.353216743s" totalImagesPullingTime="1.648593726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:28.352267481 +0000 UTC m=+57.638579999" watchObservedRunningTime="2026-09-17 18:39:28.353216743 +0000 UTC m=+57.639529260"
Sep 17 18:39:29.493337 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:29.493299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:29.493916 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:29.493454    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:29.493916 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:29.493532    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls podName:9d4f84b2-13fa-465c-bca2-aa7f8027c5f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:33.49351226 +0000 UTC m=+62.779824776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-ddvnb" (UID: "9d4f84b2-13fa-465c-bca2-aa7f8027c5f5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:30.340830 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.340799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg" event={"ID":"dcf0ad40-19d2-44f9-a69d-911b05e056e9","Type":"ContainerStarted","Data":"e6b3d6aafbe5b0c9f173b7f083f5822bc30106d060a485d776fc40d32ddf04fc"}
Sep 17 18:39:30.368563 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.368498    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-km4bg" podStartSLOduration=1.8348170750000001 podStartE2EDuration="5.368477978s" totalImagesPullingTime="3.533660903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:30.366654123 +0000 UTC m=+59.652966641" watchObservedRunningTime="2026-09-17 18:39:30.368477978 +0000 UTC m=+59.654790496"
Sep 17 18:39:30.553506 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.553469    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z2bgs"]
Sep 17 18:39:30.570584 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.570534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z2bgs"]
Sep 17 18:39:30.570737 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.570681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.574004 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.573978    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:39:30.574144 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.574008    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:39:30.574144 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.574057    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtkmh\""
Sep 17 18:39:30.574144 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.574071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:39:30.575091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.575072    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:39:30.600535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.600477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-crio-socket\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.600535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.600510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.600689 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.600571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdfx\" (UniqueName: \"kubernetes.io/projected/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-api-access-7rdfx\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.600689 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.600600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-data-volume\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.600689 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.600644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.701970 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.701933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-crio-socket\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.702132 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.701980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.702132 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.702042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdfx\" (UniqueName: \"kubernetes.io/projected/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-api-access-7rdfx\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.702132 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.702055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-crio-socket\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.702132 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.702074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-data-volume\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.702132 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.702114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.702401 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:30.702139    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:30.702401 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:30.702228    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:30.702401 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:30.702234    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:31.202211916 +0000 UTC m=+60.488524433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:30.702401 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:30.702286    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:31.202271938 +0000 UTC m=+60.488584438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:30.702597 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.702461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-data-volume\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:30.710657 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:30.710634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdfx\" (UniqueName: \"kubernetes.io/projected/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-api-access-7rdfx\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:31.206604 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:31.206564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:31.206784 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:31.206713    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:31.206784 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:31.206752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:31.206884 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:31.206789    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:32.206772969 +0000 UTC m=+61.493085464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:31.206884 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:31.206810    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:31.206990 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:31.206891    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:32.20687877 +0000 UTC m=+61.493191290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:32.212860 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:32.212821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:32.213272 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:32.212897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:32.213272 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:32.212930    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:32.213272 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:32.212988    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:34.212974639 +0000 UTC m=+63.499287134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:32.213272 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:32.213008    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:32.213272 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:32.213063    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:34.21304815 +0000 UTC m=+63.499360663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:33.274103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:33.274075    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lf85"
Sep 17 18:39:33.317696 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:33.317670    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-99thn_be519205-5d21-460c-89ed-53124a87b5fe/dns-node-resolver/0.log"
Sep 17 18:39:33.524083 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:33.524042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:33.524269 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:33.524166    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:33.524269 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:33.524219    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls podName:9d4f84b2-13fa-465c-bca2-aa7f8027c5f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:41.524206174 +0000 UTC m=+70.810518668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-ddvnb" (UID: "9d4f84b2-13fa-465c-bca2-aa7f8027c5f5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:34.119178 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:34.119151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8k74_16b01038-bf4c-4648-bf2c-b5a073762284/node-ca/0.log"
Sep 17 18:39:34.228218 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:34.228180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:34.228412 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:34.228291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:34.228412 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:34.228325    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:34.228412 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:34.228388    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:34.228577 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:34.228390    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.228371946 +0000 UTC m=+67.514684440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:34.228577 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:34.228467    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.2284319 +0000 UTC m=+67.514744395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:35.116563 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:35.116534    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpkk2_e4d4de15-fc5d-48c9-aa48-da635f088ef7/serve-healthcheck-canary/0.log"
Sep 17 18:39:38.262227 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.262187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:38.262630 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.262283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:38.262630 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.262331    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:38.262630 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.262376    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:38.262630 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.262401    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:46.262387321 +0000 UTC m=+75.548699821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:38.262630 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.262428    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:46.26241735 +0000 UTC m=+75.548729845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:38.362926 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.362900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:39:38.363067 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.362948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:39:38.363067 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.363046    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:38.363147 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.363065    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:38.363147 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.363080    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4f5d4d78-j6smq: secret "image-registry-tls" not found
Sep 17 18:39:38.363147 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.363090    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls podName:0bc865a8-e383-4521-b9ff-0c2a3c0bdbff nodeName:}" failed. No retries permitted until 2026-09-17 18:40:10.363078408 +0000 UTC m=+99.649390903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls") pod "dns-default-g9jcp" (UID: "0bc865a8-e383-4521-b9ff-0c2a3c0bdbff") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:38.363265 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.363188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls podName:e1dea3f0-743f-455e-a3f4-0d64dae18b2c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:10.363176193 +0000 UTC m=+99.649488688 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls") pod "image-registry-5d4f5d4d78-j6smq" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c") : secret "image-registry-tls" not found
Sep 17 18:39:38.766492 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.766450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:38.766643 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.766509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:39:38.771242 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.771222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:38.771309 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.771246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:38.776824 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.776805    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 18:39:38.776916 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:38.776884    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs podName:0255f4fa-4df9-439c-8cc2-5b921c8a64a1 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:42.776863388 +0000 UTC m=+132.063175895 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs") pod "network-metrics-daemon-jhxpv" (UID: "0255f4fa-4df9-439c-8cc2-5b921c8a64a1") : secret "metrics-daemon-secret" not found
Sep 17 18:39:38.778586 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.778567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/433bbf08-f013-4ad0-8fd0-f592eba5ac4c-original-pull-secret\") pod \"global-pull-secret-syncer-gvpvp\" (UID: \"433bbf08-f013-4ad0-8fd0-f592eba5ac4c\") " pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:38.867507 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.867459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:38.869988 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.869969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/d63ac213-96df-4e51-9fd9-b76edfb86467-kube-api-access-rrdxm\") pod \"network-check-target-pzcln\" (UID: \"d63ac213-96df-4e51-9fd9-b76edfb86467\") " pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:38.879076 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.879058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gk2t2\""
Sep 17 18:39:38.881196 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.881182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvpvp"
Sep 17 18:39:38.886178 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:38.886164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:39.170716 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.170648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gvpvp"]
Sep 17 18:39:39.174000 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:39.173969    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433bbf08_f013_4ad0_8fd0_f592eba5ac4c.slice/crio-6099486704e60c7092399ff2db2dd357e78bae4e6bc877954dfeaa0be42164a2 WatchSource:0}: Error finding container 6099486704e60c7092399ff2db2dd357e78bae4e6bc877954dfeaa0be42164a2: Status 404 returned error can't find the container with id 6099486704e60c7092399ff2db2dd357e78bae4e6bc877954dfeaa0be42164a2
Sep 17 18:39:39.175744 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.175716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pzcln"]
Sep 17 18:39:39.178848 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:39.178827    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63ac213_96df_4e51_9fd9_b76edfb86467.slice/crio-ad75c43607d83b559b16c663226eb9fbb5810b31bba8fee4eaf6799cd3be62fb WatchSource:0}: Error finding container ad75c43607d83b559b16c663226eb9fbb5810b31bba8fee4eaf6799cd3be62fb: Status 404 returned error can't find the container with id ad75c43607d83b559b16c663226eb9fbb5810b31bba8fee4eaf6799cd3be62fb
Sep 17 18:39:39.364456 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.364407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pzcln" event={"ID":"d63ac213-96df-4e51-9fd9-b76edfb86467","Type":"ContainerStarted","Data":"e9e849b0cac8ff56345a8affbd732e1669d0bfe142fbb7cbbed7b15d46ff715c"}
Sep 17 18:39:39.364890 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.364489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:39:39.364890 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.364509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pzcln" event={"ID":"d63ac213-96df-4e51-9fd9-b76edfb86467","Type":"ContainerStarted","Data":"ad75c43607d83b559b16c663226eb9fbb5810b31bba8fee4eaf6799cd3be62fb"}
Sep 17 18:39:39.365620 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.365597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gvpvp" event={"ID":"433bbf08-f013-4ad0-8fd0-f592eba5ac4c","Type":"ContainerStarted","Data":"6099486704e60c7092399ff2db2dd357e78bae4e6bc877954dfeaa0be42164a2"}
Sep 17 18:39:39.389981 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:39.389933    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pzcln" podStartSLOduration=66.389918954 podStartE2EDuration="1m6.389918954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:39.389494069 +0000 UTC m=+68.675806585" watchObservedRunningTime="2026-09-17 18:39:39.389918954 +0000 UTC m=+68.676231472"
Sep 17 18:39:41.592005 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:41.591967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:41.592523 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:41.592119    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:41.592523 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:41.592180    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls podName:9d4f84b2-13fa-465c-bca2-aa7f8027c5f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:57.592163416 +0000 UTC m=+86.878475923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-ddvnb" (UID: "9d4f84b2-13fa-465c-bca2-aa7f8027c5f5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:39:43.385123 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:43.385084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gvpvp" event={"ID":"433bbf08-f013-4ad0-8fd0-f592eba5ac4c","Type":"ContainerStarted","Data":"bd7873c736d23dbc734da93b3b8209a8fe93a711e58466a9604b16d72604c44f"}
Sep 17 18:39:43.401561 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:43.401498    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gvpvp" podStartSLOduration=66.385737764 podStartE2EDuration="1m10.401479823s" totalImagesPullingTime="4.015742059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:43.4013676 +0000 UTC m=+72.687680118" watchObservedRunningTime="2026-09-17 18:39:43.401479823 +0000 UTC m=+72.687792343"
Sep 17 18:39:46.328015 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:46.327901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:46.328015 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:46.327970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:46.328463 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:46.328052    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:46.328463 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:39:46.328124    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls podName:81b62802-e1c3-4cb0-9d7d-11d669fc6e32 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:02.328107206 +0000 UTC m=+91.614419703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2bgs" (UID: "81b62802-e1c3-4cb0-9d7d-11d669fc6e32") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:46.328463 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:46.328455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:39:53.903591 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.903554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"]
Sep 17 18:39:53.905939 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.905924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:53.916506 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.916485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:39:53.916797 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.916769    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:39:53.917310 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.917289    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:39:53.917515 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.917493    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:39:53.917580 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.917561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:39:53.917660 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.917619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:39:53.917704 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.917688    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:39:53.921415 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.921395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"]
Sep 17 18:39:53.923065 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.923051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:53.931777 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.931759    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 18:39:53.937916 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.937894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"]
Sep 17 18:39:53.966386 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.966358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"]
Sep 17 18:39:53.984712 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0977d704-3769-43e9-932a-e46f3ea8cd27-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:53.984834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-ca\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:53.984834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxzgq\" (UniqueName: \"kubernetes.io/projected/403034ca-f221-4a06-8e2a-111a1f7719c4-kube-api-access-vxzgq\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:53.984834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/403034ca-f221-4a06-8e2a-111a1f7719c4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:53.984834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:53.984834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kqr\" (UniqueName: \"kubernetes.io/projected/0977d704-3769-43e9-932a-e46f3ea8cd27-kube-api-access-d5kqr\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:53.985016 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977d704-3769-43e9-932a-e46f3ea8cd27-tmp\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:53.985016 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-hub\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:53.985016 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:53.984895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxzgq\" (UniqueName: \"kubernetes.io/projected/403034ca-f221-4a06-8e2a-111a1f7719c4-kube-api-access-vxzgq\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/403034ca-f221-4a06-8e2a-111a1f7719c4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kqr\" (UniqueName: \"kubernetes.io/projected/0977d704-3769-43e9-932a-e46f3ea8cd27-kube-api-access-d5kqr\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.085529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977d704-3769-43e9-932a-e46f3ea8cd27-tmp\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.085529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-hub\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085783 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0977d704-3769-43e9-932a-e46f3ea8cd27-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.085783 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-ca\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.085930 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.085905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977d704-3769-43e9-932a-e46f3ea8cd27-tmp\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.086216 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.086193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/403034ca-f221-4a06-8e2a-111a1f7719c4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.088431 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.088407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.088546 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.088417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-ca\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.088546 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.088472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0977d704-3769-43e9-932a-e46f3ea8cd27-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.088546 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.088493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-hub\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.088546 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.088499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/403034ca-f221-4a06-8e2a-111a1f7719c4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.108662 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.108641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxzgq\" (UniqueName: \"kubernetes.io/projected/403034ca-f221-4a06-8e2a-111a1f7719c4-kube-api-access-vxzgq\") pod \"cluster-proxy-proxy-agent-85b6fb559d-z9gcw\" (UID: \"403034ca-f221-4a06-8e2a-111a1f7719c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.109857 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.109836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kqr\" (UniqueName: \"kubernetes.io/projected/0977d704-3769-43e9-932a-e46f3ea8cd27-kube-api-access-d5kqr\") pod \"klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d\" (UID: \"0977d704-3769-43e9-932a-e46f3ea8cd27\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.223959 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.223929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"
Sep 17 18:39:54.231588 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.231563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:39:54.506463 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.506357    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"]
Sep 17 18:39:54.509300 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:54.509274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw"]
Sep 17 18:39:54.509832 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:54.509805    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0977d704_3769_43e9_932a_e46f3ea8cd27.slice/crio-a90e6ebf70b04f373f9688ba0239ed3c9537c607b056ae52ba0815e5887e4546 WatchSource:0}: Error finding container a90e6ebf70b04f373f9688ba0239ed3c9537c607b056ae52ba0815e5887e4546: Status 404 returned error can't find the container with id a90e6ebf70b04f373f9688ba0239ed3c9537c607b056ae52ba0815e5887e4546
Sep 17 18:39:54.514060 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:39:54.514036    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod403034ca_f221_4a06_8e2a_111a1f7719c4.slice/crio-46767d3901abf6635575d834a1e310cde01d187651638ddc24c5b75f1e67a86d WatchSource:0}: Error finding container 46767d3901abf6635575d834a1e310cde01d187651638ddc24c5b75f1e67a86d: Status 404 returned error can't find the container with id 46767d3901abf6635575d834a1e310cde01d187651638ddc24c5b75f1e67a86d
Sep 17 18:39:55.414822 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:55.414779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw" event={"ID":"403034ca-f221-4a06-8e2a-111a1f7719c4","Type":"ContainerStarted","Data":"46767d3901abf6635575d834a1e310cde01d187651638ddc24c5b75f1e67a86d"}
Sep 17 18:39:55.415867 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:55.415844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d" event={"ID":"0977d704-3769-43e9-932a-e46f3ea8cd27","Type":"ContainerStarted","Data":"a90e6ebf70b04f373f9688ba0239ed3c9537c607b056ae52ba0815e5887e4546"}
Sep 17 18:39:57.615346 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:57.615303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:57.618490 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:57.618461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d4f84b2-13fa-465c-bca2-aa7f8027c5f5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-ddvnb\" (UID: \"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:39:57.802969 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:57.802930    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ksb7k\""
Sep 17 18:39:57.811149 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:39:57.811123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"
Sep 17 18:40:00.631659 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:00.631580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb"]
Sep 17 18:40:00.634394 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:40:00.634367    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4f84b2_13fa_465c_bca2_aa7f8027c5f5.slice/crio-f0a580d62523048a30173107fa6e815613e115e0d5255fb39fcee3ce8f47a1c2 WatchSource:0}: Error finding container f0a580d62523048a30173107fa6e815613e115e0d5255fb39fcee3ce8f47a1c2: Status 404 returned error can't find the container with id f0a580d62523048a30173107fa6e815613e115e0d5255fb39fcee3ce8f47a1c2
Sep 17 18:40:01.436193 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.436157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d" event={"ID":"0977d704-3769-43e9-932a-e46f3ea8cd27","Type":"ContainerStarted","Data":"4892a8fc7e48ddb2f39e5f62063b0c91f6a5531a273944ee802ad7343671ac63"}
Sep 17 18:40:01.436392 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.436345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:40:01.438031 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.438010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d"
Sep 17 18:40:01.438469 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.438449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw" event={"ID":"403034ca-f221-4a06-8e2a-111a1f7719c4","Type":"ContainerStarted","Data":"6aa8187b761605f06cf0958f767c6c5b1ffe271fd295b0d135bc31d300218d46"}
Sep 17 18:40:01.438608 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.438486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw" event={"ID":"403034ca-f221-4a06-8e2a-111a1f7719c4","Type":"ContainerStarted","Data":"38f97e973b6f27de5104463d1ebe6fd7816561c0e7208b193dc78281fd809d6f"}
Sep 17 18:40:01.438608 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.438502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw" event={"ID":"403034ca-f221-4a06-8e2a-111a1f7719c4","Type":"ContainerStarted","Data":"ea8c776a618345cc51d6fd3b751b0f3bbb77ded747f4db48b686bec2416f333a"}
Sep 17 18:40:01.439617 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.439596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb" event={"ID":"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5","Type":"ContainerStarted","Data":"f0a580d62523048a30173107fa6e815613e115e0d5255fb39fcee3ce8f47a1c2"}
Sep 17 18:40:01.455146 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.455102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d5d9cfbbc-zjq9d" podStartSLOduration=2.578392841 podStartE2EDuration="8.455088456s" totalImagesPullingTime="5.876695615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:01.453564953 +0000 UTC m=+90.739877469" watchObservedRunningTime="2026-09-17 18:40:01.455088456 +0000 UTC m=+90.741400973"
Sep 17 18:40:01.475407 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:01.475358    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6fb559d-z9gcw" podStartSLOduration=2.604120871 podStartE2EDuration="8.475343986s" totalImagesPullingTime="5.871223115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:01.474343267 +0000 UTC m=+90.760655784" watchObservedRunningTime="2026-09-17 18:40:01.475343986 +0000 UTC m=+90.761656535"
Sep 17 18:40:02.352104 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:02.352063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:40:02.354769 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:02.354740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b62802-e1c3-4cb0-9d7d-11d669fc6e32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2bgs\" (UID: \"81b62802-e1c3-4cb0-9d7d-11d669fc6e32\") " pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:40:02.382486 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:02.382456    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtkmh\""
Sep 17 18:40:02.390576 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:02.390557    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z2bgs"
Sep 17 18:40:02.830788 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:02.830755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z2bgs"]
Sep 17 18:40:02.834889 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:40:02.834864    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b62802_e1c3_4cb0_9d7d_11d669fc6e32.slice/crio-f0c09d87bb2f0af3105bb716e41540bee7985a681c0c4bfd9d46c973090b59c2 WatchSource:0}: Error finding container f0c09d87bb2f0af3105bb716e41540bee7985a681c0c4bfd9d46c973090b59c2: Status 404 returned error can't find the container with id f0c09d87bb2f0af3105bb716e41540bee7985a681c0c4bfd9d46c973090b59c2
Sep 17 18:40:03.448496 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:03.448463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb" event={"ID":"9d4f84b2-13fa-465c-bca2-aa7f8027c5f5","Type":"ContainerStarted","Data":"907334f175ef4a68f2324638c159dabe6c98226c54eb116b2e2240c686820535"}
Sep 17 18:40:03.450107 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:03.450078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2bgs" event={"ID":"81b62802-e1c3-4cb0-9d7d-11d669fc6e32","Type":"ContainerStarted","Data":"d11aca6998aac2ef08d6cfada5eafefddd229a9d7a8447473c5b6839d0679048"}
Sep 17 18:40:03.450241 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:03.450134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2bgs" event={"ID":"81b62802-e1c3-4cb0-9d7d-11d669fc6e32","Type":"ContainerStarted","Data":"f0c09d87bb2f0af3105bb716e41540bee7985a681c0c4bfd9d46c973090b59c2"}
Sep 17 18:40:03.468574 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:03.468524    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-ddvnb" podStartSLOduration=36.522429212 podStartE2EDuration="38.468506968s" totalImagesPullingTime="1.946077756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:03.467543939 +0000 UTC m=+92.753856457" watchObservedRunningTime="2026-09-17 18:40:03.468506968 +0000 UTC m=+92.754819484"
Sep 17 18:40:04.457005 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:04.456962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2bgs" event={"ID":"81b62802-e1c3-4cb0-9d7d-11d669fc6e32","Type":"ContainerStarted","Data":"bf5997abe46eb47c76d4c5870a9b3f245998c9551ab3ce3b236c8e559af42145"}
Sep 17 18:40:05.462079 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:05.462050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2bgs" event={"ID":"81b62802-e1c3-4cb0-9d7d-11d669fc6e32","Type":"ContainerStarted","Data":"7d90ca17420449e0ed6e0f8d847322ca2956891b4f9856d6d8ba5e637082ddf5"}
Sep 17 18:40:05.482417 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:05.482374    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z2bgs" podStartSLOduration=33.326897391 podStartE2EDuration="35.482360637s" totalImagesPullingTime="2.155463246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:05.482111247 +0000 UTC m=+94.768423766" watchObservedRunningTime="2026-09-17 18:40:05.482360637 +0000 UTC m=+94.768673153"
Sep 17 18:40:10.370786 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.370757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pzcln"
Sep 17 18:40:10.414347 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.414319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:40:10.414510 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.414371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:10.416711 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.416684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bc865a8-e383-4521-b9ff-0c2a3c0bdbff-metrics-tls\") pod \"dns-default-g9jcp\" (UID: \"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff\") " pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:40:10.416820 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.416779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"image-registry-5d4f5d4d78-j6smq\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") " pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:10.433092 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.433070    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dw5r6\""
Sep 17 18:40:10.440823 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.440806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:10.464038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.464019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zttzk\""
Sep 17 18:40:10.469084 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.469065    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:40:10.717811 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.717776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4f5d4d78-j6smq"]
Sep 17 18:40:10.720836 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:40:10.720806    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dea3f0_743f_455e_a3f4_0d64dae18b2c.slice/crio-c3aabf91a9cd17f2932ab047a61662a2b9902ed3c6c3a01e1dbeabd907c6f9f8 WatchSource:0}: Error finding container c3aabf91a9cd17f2932ab047a61662a2b9902ed3c6c3a01e1dbeabd907c6f9f8: Status 404 returned error can't find the container with id c3aabf91a9cd17f2932ab047a61662a2b9902ed3c6c3a01e1dbeabd907c6f9f8
Sep 17 18:40:10.742956 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:10.742936    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g9jcp"]
Sep 17 18:40:10.751399 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:40:10.751376    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc865a8_e383_4521_b9ff_0c2a3c0bdbff.slice/crio-405b68a7da2917fdfd8a1c5b1fcb0b1ec0c2810dc20901e209e3d10059348d30 WatchSource:0}: Error finding container 405b68a7da2917fdfd8a1c5b1fcb0b1ec0c2810dc20901e209e3d10059348d30: Status 404 returned error can't find the container with id 405b68a7da2917fdfd8a1c5b1fcb0b1ec0c2810dc20901e209e3d10059348d30
Sep 17 18:40:11.483933 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:11.483893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9jcp" event={"ID":"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff","Type":"ContainerStarted","Data":"405b68a7da2917fdfd8a1c5b1fcb0b1ec0c2810dc20901e209e3d10059348d30"}
Sep 17 18:40:11.485718 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:11.485689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq" event={"ID":"e1dea3f0-743f-455e-a3f4-0d64dae18b2c","Type":"ContainerStarted","Data":"5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173"}
Sep 17 18:40:11.485861 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:11.485736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq" event={"ID":"e1dea3f0-743f-455e-a3f4-0d64dae18b2c","Type":"ContainerStarted","Data":"c3aabf91a9cd17f2932ab047a61662a2b9902ed3c6c3a01e1dbeabd907c6f9f8"}
Sep 17 18:40:11.485861 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:11.485824    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:11.513972 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:11.513884    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq" podStartSLOduration=98.513863877 podStartE2EDuration="1m38.513863877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:11.513299831 +0000 UTC m=+100.799612348" watchObservedRunningTime="2026-09-17 18:40:11.513863877 +0000 UTC m=+100.800176398"
Sep 17 18:40:13.494476 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:13.494416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9jcp" event={"ID":"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff","Type":"ContainerStarted","Data":"0f531613b5824c1d22ffc1296710d1de333c423b211a03ee0b792254ade5905b"}
Sep 17 18:40:13.494837 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:13.494501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9jcp" event={"ID":"0bc865a8-e383-4521-b9ff-0c2a3c0bdbff","Type":"ContainerStarted","Data":"39dcdffcf59f14e79cbb56f834e7c572e8ca37e9e2a19070b8244caba5cd33a7"}
Sep 17 18:40:13.494837 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:13.494540    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:40:13.527349 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:13.527306    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g9jcp" podStartSLOduration=65.683754505 podStartE2EDuration="1m7.527291859s" totalImagesPullingTime="1.843537354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:13.527012325 +0000 UTC m=+102.813324842" watchObservedRunningTime="2026-09-17 18:40:13.527291859 +0000 UTC m=+102.813604419"
Sep 17 18:40:18.634364 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:18.634329    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4f5d4d78-j6smq"]
Sep 17 18:40:23.500865 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.500832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g9jcp"
Sep 17 18:40:23.580991 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.580947    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dtbbg"]
Sep 17 18:40:23.583693 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.583670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.586390 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.586365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:40:23.586622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.586600    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:40:23.586718 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.586650    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:40:23.586718 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.586664    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:40:23.587039 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.587019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwb4x\""
Sep 17 18:40:23.713792 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-textfile\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.713792 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.713997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-metrics-client-ca\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.713997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-root\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.713997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4prt\" (UniqueName: \"kubernetes.io/projected/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-kube-api-access-l4prt\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.713997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-sys\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.713997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.713996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.714150 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.714019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-tls\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.714150 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.714040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-wtmp\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814733 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-textfile\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814733 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814733 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-metrics-client-ca\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814998 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-root\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814998 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-root\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814998 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4prt\" (UniqueName: \"kubernetes.io/projected/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-kube-api-access-l4prt\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814998 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-sys\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814998 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.814998 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.814990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-tls\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.815265 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.815014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-sys\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.815265 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.815015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-wtmp\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.815265 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.815060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-textfile\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.815265 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.815144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-wtmp\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.815387 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.815327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-metrics-client-ca\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.815456 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.815415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.817179 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.817154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.817287 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.817233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-node-exporter-tls\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.824331 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.824303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4prt\" (UniqueName: \"kubernetes.io/projected/d1d5c0d8-4c6a-4cb6-85f5-342477ef2018-kube-api-access-l4prt\") pod \"node-exporter-dtbbg\" (UID: \"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018\") " pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.893240 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:23.893189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dtbbg"
Sep 17 18:40:23.902143 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:40:23.902115    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d5c0d8_4c6a_4cb6_85f5_342477ef2018.slice/crio-c373a5699280dcc8bbe862f81e0f9e156ddea5bf818134522b1b4e42bad53a19 WatchSource:0}: Error finding container c373a5699280dcc8bbe862f81e0f9e156ddea5bf818134522b1b4e42bad53a19: Status 404 returned error can't find the container with id c373a5699280dcc8bbe862f81e0f9e156ddea5bf818134522b1b4e42bad53a19
Sep 17 18:40:24.531694 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:24.531660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtbbg" event={"ID":"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018","Type":"ContainerStarted","Data":"c373a5699280dcc8bbe862f81e0f9e156ddea5bf818134522b1b4e42bad53a19"}
Sep 17 18:40:25.535751 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:25.535723    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1d5c0d8-4c6a-4cb6-85f5-342477ef2018" containerID="c4580ef06475023cfa2449cbaa7ac356a02b15c092ce77f8e158428916f06cad" exitCode=0
Sep 17 18:40:25.536189 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:25.535822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtbbg" event={"ID":"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018","Type":"ContainerDied","Data":"c4580ef06475023cfa2449cbaa7ac356a02b15c092ce77f8e158428916f06cad"}
Sep 17 18:40:26.541488 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:26.541456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtbbg" event={"ID":"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018","Type":"ContainerStarted","Data":"f934a186d4e27e91399a1cd6c3b2672a4248721bace802539aca49a1e2a12e16"}
Sep 17 18:40:26.541950 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:26.541517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtbbg" event={"ID":"d1d5c0d8-4c6a-4cb6-85f5-342477ef2018","Type":"ContainerStarted","Data":"191743534e365a90f54007b5323b78464060e4c14301a1f85ef6aec9abda427c"}
Sep 17 18:40:26.567987 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:26.567932    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dtbbg" podStartSLOduration=2.409043552 podStartE2EDuration="3.567918217s" totalImagesPullingTime="1.097563437s" totalInitContainerRuntime="61.311228ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:26.566063293 +0000 UTC m=+115.852375811" watchObservedRunningTime="2026-09-17 18:40:26.567918217 +0000 UTC m=+115.854230733"
Sep 17 18:40:28.640382 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:28.640358    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:42.849545 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:42.849509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:40:42.851840 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:42.851810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0255f4fa-4df9-439c-8cc2-5b921c8a64a1-metrics-certs\") pod \"network-metrics-daemon-jhxpv\" (UID: \"0255f4fa-4df9-439c-8cc2-5b921c8a64a1\") " pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:40:43.071927 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.071901    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kfmzw\""
Sep 17 18:40:43.079212 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.079195    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhxpv"
Sep 17 18:40:43.345026 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.344996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhxpv"]
Sep 17 18:40:43.347805 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:40:43.347779    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0255f4fa_4df9_439c_8cc2_5b921c8a64a1.slice/crio-bb08d6be19e47b1b28c6026b9e774824bef0f70441912c21fca9fea81bbb298e WatchSource:0}: Error finding container bb08d6be19e47b1b28c6026b9e774824bef0f70441912c21fca9fea81bbb298e: Status 404 returned error can't find the container with id bb08d6be19e47b1b28c6026b9e774824bef0f70441912c21fca9fea81bbb298e
Sep 17 18:40:43.597733 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.597696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhxpv" event={"ID":"0255f4fa-4df9-439c-8cc2-5b921c8a64a1","Type":"ContainerStarted","Data":"bb08d6be19e47b1b28c6026b9e774824bef0f70441912c21fca9fea81bbb298e"}
Sep 17 18:40:43.652886 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.652841    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq" podUID="e1dea3f0-743f-455e-a3f4-0d64dae18b2c" containerName="registry" containerID="cri-o://5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173" gracePeriod=30
Sep 17 18:40:43.914036 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.914011    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:43.956034 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-certificates\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956161 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-trusted-ca\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956161 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956161 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-image-registry-private-configuration\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956321 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956284    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-installation-pull-secrets\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956375 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-ca-trust-extracted\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956429 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956375    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpvpx\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-kube-api-access-xpvpx\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956501 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-bound-sa-token\") pod \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\" (UID: \"e1dea3f0-743f-455e-a3f4-0d64dae18b2c\") "
Sep 17 18:40:43.956560 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:43.957957 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956679    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-trusted-ca\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:43.957957 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.956843    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:43.959580 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.959377    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-kube-api-access-xpvpx" (OuterVolumeSpecName: "kube-api-access-xpvpx") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "kube-api-access-xpvpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:43.959580 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.959553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:43.959926 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.959887    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:43.959926 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.959908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:43.960825 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.960795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:43.967080 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:43.967050    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e1dea3f0-743f-455e-a3f4-0d64dae18b2c" (UID: "e1dea3f0-743f-455e-a3f4-0d64dae18b2c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:44.057257 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057230    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-bound-sa-token\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.057257 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057255    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-certificates\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.057257 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057265    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-registry-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.057453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057274    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-image-registry-private-configuration\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.057453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057284    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-installation-pull-secrets\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.057453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057293    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-ca-trust-extracted\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.057453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.057301    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpvpx\" (UniqueName: \"kubernetes.io/projected/e1dea3f0-743f-455e-a3f4-0d64dae18b2c-kube-api-access-xpvpx\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:40:44.603081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.602987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhxpv" event={"ID":"0255f4fa-4df9-439c-8cc2-5b921c8a64a1","Type":"ContainerStarted","Data":"7f04bd7162f05f774f31ecd2c7c8f926d13d69672d0d9002e92ee89907c0269e"}
Sep 17 18:40:44.603081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.603046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhxpv" event={"ID":"0255f4fa-4df9-439c-8cc2-5b921c8a64a1","Type":"ContainerStarted","Data":"2555d3457881afc002dde50ff98bdd43f9f42f345227540e52fe85dbc7d85de7"}
Sep 17 18:40:44.604240 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.604214    2565 generic.go:356] "Generic (PLEG): container finished" podID="e1dea3f0-743f-455e-a3f4-0d64dae18b2c" containerID="5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173" exitCode=0
Sep 17 18:40:44.604354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.604245    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq"
Sep 17 18:40:44.604354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.604256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq" event={"ID":"e1dea3f0-743f-455e-a3f4-0d64dae18b2c","Type":"ContainerDied","Data":"5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173"}
Sep 17 18:40:44.604354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.604283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4f5d4d78-j6smq" event={"ID":"e1dea3f0-743f-455e-a3f4-0d64dae18b2c","Type":"ContainerDied","Data":"c3aabf91a9cd17f2932ab047a61662a2b9902ed3c6c3a01e1dbeabd907c6f9f8"}
Sep 17 18:40:44.604354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.604303    2565 scope.go:117] "RemoveContainer" containerID="5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173"
Sep 17 18:40:44.618362 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.618342    2565 scope.go:117] "RemoveContainer" containerID="5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173"
Sep 17 18:40:44.618653 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:40:44.618630    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173\": container with ID starting with 5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173 not found: ID does not exist" containerID="5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173"
Sep 17 18:40:44.618734 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.618664    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173"} err="failed to get container status \"5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173\": rpc error: code = NotFound desc = could not find container \"5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173\": container with ID starting with 5239c6869888d66a48ac3aa8ae0b1ab2f4b19d6f157f8ee2485c4f6a90649173 not found: ID does not exist"
Sep 17 18:40:44.620575 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.620528    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jhxpv" podStartSLOduration=130.680868376 podStartE2EDuration="2m11.62051426s" totalImagesPullingTime="939.645884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:44.620511457 +0000 UTC m=+133.906823975" watchObservedRunningTime="2026-09-17 18:40:44.62051426 +0000 UTC m=+133.906826777"
Sep 17 18:40:44.634709 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.634686    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4f5d4d78-j6smq"]
Sep 17 18:40:44.639615 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:44.639596    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d4f5d4d78-j6smq"]
Sep 17 18:40:45.061518 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:40:45.061491    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1dea3f0-743f-455e-a3f4-0d64dae18b2c" path="/var/lib/kubelet/pods/e1dea3f0-743f-455e-a3f4-0d64dae18b2c/volumes"
Sep 17 18:43:32.991737 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:43:32.991705    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:46:50.416102 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.416072    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"]
Sep 17 18:46:50.416578 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.416295    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1dea3f0-743f-455e-a3f4-0d64dae18b2c" containerName="registry"
Sep 17 18:46:50.416578 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.416305    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1dea3f0-743f-455e-a3f4-0d64dae18b2c" containerName="registry"
Sep 17 18:46:50.416578 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.416347    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1dea3f0-743f-455e-a3f4-0d64dae18b2c" containerName="registry"
Sep 17 18:46:50.419127 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.419109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.422245 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.422220    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:46:50.422369 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.422270    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:46:50.423375 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.423358    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nf7s6\""
Sep 17 18:46:50.437402 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.437373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"]
Sep 17 18:46:50.540131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.540102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8gr\" (UniqueName: \"kubernetes.io/projected/ff502853-e366-4f40-8a6f-dc9694ab2089-kube-api-access-8w8gr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.540131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.540132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.540316 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.540165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.640446 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.640413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8gr\" (UniqueName: \"kubernetes.io/projected/ff502853-e366-4f40-8a6f-dc9694ab2089-kube-api-access-8w8gr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.640540 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.640465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.640540 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.640494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.640810 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.640793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.640847 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.640829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.651135 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.651107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8gr\" (UniqueName: \"kubernetes.io/projected/ff502853-e366-4f40-8a6f-dc9694ab2089-kube-api-access-8w8gr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:50.728046 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:50.728027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:46:51.001475 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:51.001379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"]
Sep 17 18:46:51.005384 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:46:51.005358    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff502853_e366_4f40_8a6f_dc9694ab2089.slice/crio-3bd156eaa40cae16bc2113e47e76bfaa4b1b5e9f858e848206a3c2a01be34395 WatchSource:0}: Error finding container 3bd156eaa40cae16bc2113e47e76bfaa4b1b5e9f858e848206a3c2a01be34395: Status 404 returned error can't find the container with id 3bd156eaa40cae16bc2113e47e76bfaa4b1b5e9f858e848206a3c2a01be34395
Sep 17 18:46:51.007336 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:51.007320    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:46:51.796042 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:51.796003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx" event={"ID":"ff502853-e366-4f40-8a6f-dc9694ab2089","Type":"ContainerStarted","Data":"3bd156eaa40cae16bc2113e47e76bfaa4b1b5e9f858e848206a3c2a01be34395"}
Sep 17 18:46:58.823222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:58.823187    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerID="3b67150a90b6256ca8ded4929d0149da2e71e6ec7b2f5653f977eb1a9b2993fd" exitCode=0
Sep 17 18:46:58.823622 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:46:58.823274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx" event={"ID":"ff502853-e366-4f40-8a6f-dc9694ab2089","Type":"ContainerDied","Data":"3b67150a90b6256ca8ded4929d0149da2e71e6ec7b2f5653f977eb1a9b2993fd"}
Sep 17 18:47:10.864546 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:10.864519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx" event={"ID":"ff502853-e366-4f40-8a6f-dc9694ab2089","Type":"ContainerStarted","Data":"d2d0e60b35dc1b47be71c29445fd4ecf20556eb3a71d25c90aea9bb69eaf7f9b"}
Sep 17 18:47:11.869643 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:11.869601    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerID="d2d0e60b35dc1b47be71c29445fd4ecf20556eb3a71d25c90aea9bb69eaf7f9b" exitCode=0
Sep 17 18:47:11.870021 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:11.869682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx" event={"ID":"ff502853-e366-4f40-8a6f-dc9694ab2089","Type":"ContainerDied","Data":"d2d0e60b35dc1b47be71c29445fd4ecf20556eb3a71d25c90aea9bb69eaf7f9b"}
Sep 17 18:47:19.903018 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:19.902973    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerID="2a75d0f15cd8ef75aa4e493a2c8198ca2f3208c2e969dbb58f96d42d48b3fdee" exitCode=0
Sep 17 18:47:19.903475 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:19.903062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx" event={"ID":"ff502853-e366-4f40-8a6f-dc9694ab2089","Type":"ContainerDied","Data":"2a75d0f15cd8ef75aa4e493a2c8198ca2f3208c2e969dbb58f96d42d48b3fdee"}
Sep 17 18:47:21.025577 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.025544    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:47:21.154047 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.154011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-util\") pod \"ff502853-e366-4f40-8a6f-dc9694ab2089\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") "
Sep 17 18:47:21.154250 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.154100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w8gr\" (UniqueName: \"kubernetes.io/projected/ff502853-e366-4f40-8a6f-dc9694ab2089-kube-api-access-8w8gr\") pod \"ff502853-e366-4f40-8a6f-dc9694ab2089\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") "
Sep 17 18:47:21.154250 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.154124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-bundle\") pod \"ff502853-e366-4f40-8a6f-dc9694ab2089\" (UID: \"ff502853-e366-4f40-8a6f-dc9694ab2089\") "
Sep 17 18:47:21.154797 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.154773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-bundle" (OuterVolumeSpecName: "bundle") pod "ff502853-e366-4f40-8a6f-dc9694ab2089" (UID: "ff502853-e366-4f40-8a6f-dc9694ab2089"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:21.156369 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.156330    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff502853-e366-4f40-8a6f-dc9694ab2089-kube-api-access-8w8gr" (OuterVolumeSpecName: "kube-api-access-8w8gr") pod "ff502853-e366-4f40-8a6f-dc9694ab2089" (UID: "ff502853-e366-4f40-8a6f-dc9694ab2089"). InnerVolumeSpecName "kube-api-access-8w8gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:21.158236 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.158193    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-util" (OuterVolumeSpecName: "util") pod "ff502853-e366-4f40-8a6f-dc9694ab2089" (UID: "ff502853-e366-4f40-8a6f-dc9694ab2089"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:21.254997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.254970    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8gr\" (UniqueName: \"kubernetes.io/projected/ff502853-e366-4f40-8a6f-dc9694ab2089-kube-api-access-8w8gr\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:47:21.254997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.254994    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-bundle\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:47:21.254997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.255004    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff502853-e366-4f40-8a6f-dc9694ab2089-util\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:47:21.912018 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.911988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx" event={"ID":"ff502853-e366-4f40-8a6f-dc9694ab2089","Type":"ContainerDied","Data":"3bd156eaa40cae16bc2113e47e76bfaa4b1b5e9f858e848206a3c2a01be34395"}
Sep 17 18:47:21.912194 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.912030    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asdmdx"
Sep 17 18:47:21.912194 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:21.912029    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bd156eaa40cae16bc2113e47e76bfaa4b1b5e9f858e848206a3c2a01be34395"
Sep 17 18:47:36.325800 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.325765    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-s4wxx"]
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.325986    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="extract"
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.325996    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="extract"
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.326007    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="util"
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.326012    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="util"
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.326022    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="pull"
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.326027    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="pull"
Sep 17 18:47:36.326222 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.326071    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff502853-e366-4f40-8a6f-dc9694ab2089" containerName="extract"
Sep 17 18:47:36.328988 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.328969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.332692 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.332673    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:47:36.333106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.333092    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:47:36.333680 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.333657    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-sqlct\""
Sep 17 18:47:36.333804 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.333687    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:47:36.333804 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.333695    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:47:36.333804 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.333660    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:47:36.346505 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.346484    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-s4wxx"]
Sep 17 18:47:36.459010 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.458980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hszb\" (UniqueName: \"kubernetes.io/projected/e258bb56-58b6-4ab9-8752-43eb0522bf46-kube-api-access-2hszb\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.459131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.459017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e258bb56-58b6-4ab9-8752-43eb0522bf46-certificates\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.459131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.459048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e258bb56-58b6-4ab9-8752-43eb0522bf46-cabundle0\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.560212 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.560192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hszb\" (UniqueName: \"kubernetes.io/projected/e258bb56-58b6-4ab9-8752-43eb0522bf46-kube-api-access-2hszb\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.560323 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.560220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e258bb56-58b6-4ab9-8752-43eb0522bf46-certificates\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.560323 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.560240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e258bb56-58b6-4ab9-8752-43eb0522bf46-cabundle0\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.560836 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.560813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e258bb56-58b6-4ab9-8752-43eb0522bf46-cabundle0\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.562630 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.562607    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e258bb56-58b6-4ab9-8752-43eb0522bf46-certificates\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.570391 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.570373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hszb\" (UniqueName: \"kubernetes.io/projected/e258bb56-58b6-4ab9-8752-43eb0522bf46-kube-api-access-2hszb\") pod \"keda-operator-579c8958f7-s4wxx\" (UID: \"e258bb56-58b6-4ab9-8752-43eb0522bf46\") " pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.638797 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.638744    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:36.802746 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.802715    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"]
Sep 17 18:47:36.804611 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.804597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:36.809092 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.808689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:47:36.858133 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.858093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"]
Sep 17 18:47:36.927556 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.927492    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-s4wxx"]
Sep 17 18:47:36.931167 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:47:36.931139    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode258bb56_58b6_4ab9_8752_43eb0522bf46.slice/crio-c2a9aa29f1082dd2473c260bbc082cb0d19e697cab277ad13445fc07978de04a WatchSource:0}: Error finding container c2a9aa29f1082dd2473c260bbc082cb0d19e697cab277ad13445fc07978de04a: Status 404 returned error can't find the container with id c2a9aa29f1082dd2473c260bbc082cb0d19e697cab277ad13445fc07978de04a
Sep 17 18:47:36.963621 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.963589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:36.963712 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.963633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf97p\" (UniqueName: \"kubernetes.io/projected/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-kube-api-access-lf97p\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:36.963712 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.963676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-certificates\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:36.964590 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:36.964569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-s4wxx" event={"ID":"e258bb56-58b6-4ab9-8752-43eb0522bf46","Type":"ContainerStarted","Data":"c2a9aa29f1082dd2473c260bbc082cb0d19e697cab277ad13445fc07978de04a"}
Sep 17 18:47:37.064321 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.064297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.064463 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.064337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf97p\" (UniqueName: \"kubernetes.io/projected/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-kube-api-access-lf97p\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.064533 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.064470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-certificates\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.064704 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.064685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.066834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.066816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-certificates\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.082394 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.082375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf97p\" (UniqueName: \"kubernetes.io/projected/749a6234-0dbe-4f0f-a9f6-5d23fcb70c38-kube-api-access-lf97p\") pod \"keda-metrics-apiserver-77857b7b98-krzs7\" (UID: \"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.117421 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.117399    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:37.414879 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.414853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"]
Sep 17 18:47:37.416897 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:47:37.416871    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749a6234_0dbe_4f0f_a9f6_5d23fcb70c38.slice/crio-26fbd0f3e3747e3ad83090c76702a011da2e9f0b26adedca900076bec4717805 WatchSource:0}: Error finding container 26fbd0f3e3747e3ad83090c76702a011da2e9f0b26adedca900076bec4717805: Status 404 returned error can't find the container with id 26fbd0f3e3747e3ad83090c76702a011da2e9f0b26adedca900076bec4717805
Sep 17 18:47:37.971088 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:37.971042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7" event={"ID":"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38","Type":"ContainerStarted","Data":"26fbd0f3e3747e3ad83090c76702a011da2e9f0b26adedca900076bec4717805"}
Sep 17 18:47:40.982794 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:40.982754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-s4wxx" event={"ID":"e258bb56-58b6-4ab9-8752-43eb0522bf46","Type":"ContainerStarted","Data":"b9ee670d4f3c000e233fb2dfc8459571ca3a49f26493a4e47f5348612ce86f6e"}
Sep 17 18:47:40.983241 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:40.982961    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:47:40.984370 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:40.984344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7" event={"ID":"749a6234-0dbe-4f0f-a9f6-5d23fcb70c38","Type":"ContainerStarted","Data":"15f581bc09416f27f74eb8ae782eee8b1dbc1138f0e550efa11826f211c61cc1"}
Sep 17 18:47:40.984506 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:40.984394    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:47:41.000325 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:41.000281    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-s4wxx" podStartSLOduration=1.136755036 podStartE2EDuration="5.000269199s" totalImagesPullingTime="3.863514163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:40.999023161 +0000 UTC m=+550.285335679" watchObservedRunningTime="2026-09-17 18:47:41.000269199 +0000 UTC m=+550.286581716"
Sep 17 18:47:41.016288 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:41.016249    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7" podStartSLOduration=1.643587048 podStartE2EDuration="5.016235387s" totalImagesPullingTime="3.372648339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:41.015352585 +0000 UTC m=+550.301665107" watchObservedRunningTime="2026-09-17 18:47:41.016235387 +0000 UTC m=+550.302547903"
Sep 17 18:47:51.993312 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:47:51.993286    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krzs7"
Sep 17 18:48:01.991582 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:48:01.991557    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-s4wxx"
Sep 17 18:49:06.638401 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.638370    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-s6t58"]
Sep 17 18:49:06.641328 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.641309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.644198 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.644179    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:49:06.644305 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.644287    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:49:06.644999 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.644986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9qh76\""
Sep 17 18:49:06.645076 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.644986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:49:06.663893 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.663873    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-s6t58"]
Sep 17 18:49:06.756343 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.756319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a4b1d10-d492-40eb-a0f7-c8f49875f5c9-data\") pod \"seaweedfs-64568bcd49-s6t58\" (UID: \"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9\") " pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.756460 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.756361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tsd4\" (UniqueName: \"kubernetes.io/projected/4a4b1d10-d492-40eb-a0f7-c8f49875f5c9-kube-api-access-9tsd4\") pod \"seaweedfs-64568bcd49-s6t58\" (UID: \"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9\") " pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.856987 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.856964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a4b1d10-d492-40eb-a0f7-c8f49875f5c9-data\") pod \"seaweedfs-64568bcd49-s6t58\" (UID: \"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9\") " pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.857080 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.857005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tsd4\" (UniqueName: \"kubernetes.io/projected/4a4b1d10-d492-40eb-a0f7-c8f49875f5c9-kube-api-access-9tsd4\") pod \"seaweedfs-64568bcd49-s6t58\" (UID: \"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9\") " pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.857330 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.857310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a4b1d10-d492-40eb-a0f7-c8f49875f5c9-data\") pod \"seaweedfs-64568bcd49-s6t58\" (UID: \"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9\") " pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.867414 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.867393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tsd4\" (UniqueName: \"kubernetes.io/projected/4a4b1d10-d492-40eb-a0f7-c8f49875f5c9-kube-api-access-9tsd4\") pod \"seaweedfs-64568bcd49-s6t58\" (UID: \"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9\") " pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:06.950493 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:06.950476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:07.223404 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:07.223336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-s6t58"]
Sep 17 18:49:07.226103 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:49:07.226072    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4b1d10_d492_40eb_a0f7_c8f49875f5c9.slice/crio-a0b7a308ba7f6c1f850f71a0f46e079cbe73bef5ac9dc07d36c778c6773e4d2f WatchSource:0}: Error finding container a0b7a308ba7f6c1f850f71a0f46e079cbe73bef5ac9dc07d36c778c6773e4d2f: Status 404 returned error can't find the container with id a0b7a308ba7f6c1f850f71a0f46e079cbe73bef5ac9dc07d36c778c6773e4d2f
Sep 17 18:49:07.283184 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:07.283157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-s6t58" event={"ID":"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9","Type":"ContainerStarted","Data":"a0b7a308ba7f6c1f850f71a0f46e079cbe73bef5ac9dc07d36c778c6773e4d2f"}
Sep 17 18:49:10.296284 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:10.296255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-s6t58" event={"ID":"4a4b1d10-d492-40eb-a0f7-c8f49875f5c9","Type":"ContainerStarted","Data":"9ddf03aa8aa65bf07bed2cc092103cdf43e7b53c4dd381035bf250da097679d9"}
Sep 17 18:49:10.296710 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:10.296375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:49:10.314858 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:10.314814    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-s6t58" podStartSLOduration=1.463755702 podStartE2EDuration="4.314799321s" totalImagesPullingTime="2.851043619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:10.314236654 +0000 UTC m=+639.600549172" watchObservedRunningTime="2026-09-17 18:49:10.314799321 +0000 UTC m=+639.601111892"
Sep 17 18:49:16.301865 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:49:16.301835    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-s6t58"
Sep 17 18:50:25.395938 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.395867    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-txl5t"]
Sep 17 18:50:25.398810 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.398792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.401882 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.401862    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-p76fr\""
Sep 17 18:50:25.401974 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.401862    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:50:25.406372 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.406352    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-jqjjn"]
Sep 17 18:50:25.409106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.409091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.411143 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.411124    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:50:25.411231 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.411187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:50:25.411231 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.411209    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-z7bhz\""
Sep 17 18:50:25.414400 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.414381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-txl5t"]
Sep 17 18:50:25.423103 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.423082    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-jqjjn"]
Sep 17 18:50:25.550513 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.550489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5556a267-b158-42be-bbcc-c5f28211c3fc-cert\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.550652 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.550527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t792m\" (UniqueName: \"kubernetes.io/projected/76adb496-0ce6-4d02-9c8d-644b0608707a-kube-api-access-t792m\") pod \"model-serving-api-7b954ffc95-txl5t\" (UID: \"76adb496-0ce6-4d02-9c8d-644b0608707a\") " pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.550652 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.550562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5556a267-b158-42be-bbcc-c5f28211c3fc-metrics-certs\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.550652 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.550602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76adb496-0ce6-4d02-9c8d-644b0608707a-tls-certs\") pod \"model-serving-api-7b954ffc95-txl5t\" (UID: \"76adb496-0ce6-4d02-9c8d-644b0608707a\") " pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.550652 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.550647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcs5\" (UniqueName: \"kubernetes.io/projected/5556a267-b158-42be-bbcc-c5f28211c3fc-kube-api-access-fxcs5\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.650980 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.650923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5556a267-b158-42be-bbcc-c5f28211c3fc-cert\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.650980 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.650955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t792m\" (UniqueName: \"kubernetes.io/projected/76adb496-0ce6-4d02-9c8d-644b0608707a-kube-api-access-t792m\") pod \"model-serving-api-7b954ffc95-txl5t\" (UID: \"76adb496-0ce6-4d02-9c8d-644b0608707a\") " pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.650980 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.650975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5556a267-b158-42be-bbcc-c5f28211c3fc-metrics-certs\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.651154 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.650992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76adb496-0ce6-4d02-9c8d-644b0608707a-tls-certs\") pod \"model-serving-api-7b954ffc95-txl5t\" (UID: \"76adb496-0ce6-4d02-9c8d-644b0608707a\") " pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.651154 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.651014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcs5\" (UniqueName: \"kubernetes.io/projected/5556a267-b158-42be-bbcc-c5f28211c3fc-kube-api-access-fxcs5\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.653298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.653279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5556a267-b158-42be-bbcc-c5f28211c3fc-metrics-certs\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.653388 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.653342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5556a267-b158-42be-bbcc-c5f28211c3fc-cert\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.653480 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.653460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76adb496-0ce6-4d02-9c8d-644b0608707a-tls-certs\") pod \"model-serving-api-7b954ffc95-txl5t\" (UID: \"76adb496-0ce6-4d02-9c8d-644b0608707a\") " pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.661873 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.661857    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxcs5\" (UniqueName: \"kubernetes.io/projected/5556a267-b158-42be-bbcc-c5f28211c3fc-kube-api-access-fxcs5\") pod \"odh-model-controller-7b966d7984-jqjjn\" (UID: \"5556a267-b158-42be-bbcc-c5f28211c3fc\") " pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.662162 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.662140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t792m\" (UniqueName: \"kubernetes.io/projected/76adb496-0ce6-4d02-9c8d-644b0608707a-kube-api-access-t792m\") pod \"model-serving-api-7b954ffc95-txl5t\" (UID: \"76adb496-0ce6-4d02-9c8d-644b0608707a\") " pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.708913 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.708891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:25.718791 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.718772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:25.988498 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.988474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-txl5t"]
Sep 17 18:50:25.991205 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:25.991182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-jqjjn"]
Sep 17 18:50:25.991573 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:50:25.991489    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76adb496_0ce6_4d02_9c8d_644b0608707a.slice/crio-93ecf8d170667aaf987cc335c4679aa3599f0dda9d662cb8a2283f4321efa7e4 WatchSource:0}: Error finding container 93ecf8d170667aaf987cc335c4679aa3599f0dda9d662cb8a2283f4321efa7e4: Status 404 returned error can't find the container with id 93ecf8d170667aaf987cc335c4679aa3599f0dda9d662cb8a2283f4321efa7e4
Sep 17 18:50:25.994995 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:50:25.994972    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5556a267_b158_42be_bbcc_c5f28211c3fc.slice/crio-41510356faf99d41d968bad5ad12492d165dc0f6476b2a3766599d08b664f81a WatchSource:0}: Error finding container 41510356faf99d41d968bad5ad12492d165dc0f6476b2a3766599d08b664f81a: Status 404 returned error can't find the container with id 41510356faf99d41d968bad5ad12492d165dc0f6476b2a3766599d08b664f81a
Sep 17 18:50:26.548821 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:26.548782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-jqjjn" event={"ID":"5556a267-b158-42be-bbcc-c5f28211c3fc","Type":"ContainerStarted","Data":"41510356faf99d41d968bad5ad12492d165dc0f6476b2a3766599d08b664f81a"}
Sep 17 18:50:26.550448 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:26.550404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-txl5t" event={"ID":"76adb496-0ce6-4d02-9c8d-644b0608707a","Type":"ContainerStarted","Data":"93ecf8d170667aaf987cc335c4679aa3599f0dda9d662cb8a2283f4321efa7e4"}
Sep 17 18:50:29.563608 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:29.563510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-jqjjn" event={"ID":"5556a267-b158-42be-bbcc-c5f28211c3fc","Type":"ContainerStarted","Data":"9730d41e6f320bba0a11ccf8a95fcdf1de24dcba071e7f89b73872c4fb6c87b4"}
Sep 17 18:50:29.563608 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:29.563582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:29.564877 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:29.564849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-txl5t" event={"ID":"76adb496-0ce6-4d02-9c8d-644b0608707a","Type":"ContainerStarted","Data":"fdc2b9f671f943e9d8a82b71a128e8f6f3b804e001815faf70593e98df0c4b72"}
Sep 17 18:50:29.564990 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:29.564898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:50:29.583521 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:29.583473    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-jqjjn" podStartSLOduration=1.226609323 podStartE2EDuration="4.583456184s" totalImagesPullingTime="3.356846861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:29.582295928 +0000 UTC m=+718.868608443" watchObservedRunningTime="2026-09-17 18:50:29.583456184 +0000 UTC m=+718.869768701"
Sep 17 18:50:29.601741 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:29.601684    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-txl5t" podStartSLOduration=1.2867030449999999 podStartE2EDuration="4.601668667s" totalImagesPullingTime="3.314965622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:29.600729484 +0000 UTC m=+718.887041999" watchObservedRunningTime="2026-09-17 18:50:29.601668667 +0000 UTC m=+718.887981183"
Sep 17 18:50:40.570616 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:40.570583    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-jqjjn"
Sep 17 18:50:40.572415 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:50:40.572392    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-txl5t"
Sep 17 18:52:12.779444 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.779399    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"]
Sep 17 18:52:12.782753 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.782734    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.785172 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.785149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:52:12.785270 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.785253    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:52:12.785270 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.785263    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:52:12.786270 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.785990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:52:12.790460 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.786662    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 18:52:12.790460 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.786986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:52:12.790460 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.787067    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:52:12.793811 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.793788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"]
Sep 17 18:52:12.812669 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.812639    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"]
Sep 17 18:52:12.815875 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.815860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:12.818003 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.817985    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fe81c-predictor-serving-cert\""
Sep 17 18:52:12.818081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.818026    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fe81c-kube-rbac-proxy-sar-config\""
Sep 17 18:52:12.829060 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.829035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"]
Sep 17 18:52:12.892425 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.892398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktr4\" (UniqueName: \"kubernetes.io/projected/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-api-access-pktr4\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.892572 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.892428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df56d2a-69ae-4d50-b72f-260b00657bdf-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.892572 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.892536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df56d2a-69ae-4d50-b72f-260b00657bdf-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.892572 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.892565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.892715 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.892636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.993413 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.993413 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:12.993639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:12.993639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktr4\" (UniqueName: \"kubernetes.io/projected/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-api-access-pktr4\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.993639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df56d2a-69ae-4d50-b72f-260b00657bdf-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.993639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkr8m\" (UniqueName: \"kubernetes.io/projected/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-api-access-nkr8m\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:12.993639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:12.993899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:12.993899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df56d2a-69ae-4d50-b72f-260b00657bdf-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.993899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.994038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.993963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df56d2a-69ae-4d50-b72f-260b00657bdf-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.994038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.994015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.994286 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.994270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:12.996047 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:12.996018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df56d2a-69ae-4d50-b72f-260b00657bdf-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:13.000686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.000668    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktr4\" (UniqueName: \"kubernetes.io/projected/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-api-access-pktr4\") pod \"isvc-canary-rollback-predictor-69978998f6-gd8xc\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:13.094938 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.094869    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:13.095054 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.094990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkr8m\" (UniqueName: \"kubernetes.io/projected/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-api-access-nkr8m\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095054 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095054 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095216 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095216 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095405 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095752 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.095822 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.095785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.097409 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.097389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.103068 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.103043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkr8m\" (UniqueName: \"kubernetes.io/projected/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-api-access-nkr8m\") pod \"isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.125007 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.124981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:13.370362 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.370322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"]
Sep 17 18:52:13.372761 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:52:13.372738    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df56d2a_69ae_4d50_b72f_260b00657bdf.slice/crio-bc44e91d454f950c55fe88143730b5222f077f72dd7f6eb8d812defa14d226a2 WatchSource:0}: Error finding container bc44e91d454f950c55fe88143730b5222f077f72dd7f6eb8d812defa14d226a2: Status 404 returned error can't find the container with id bc44e91d454f950c55fe88143730b5222f077f72dd7f6eb8d812defa14d226a2
Sep 17 18:52:13.374924 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.374908    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:52:13.404551 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.404490    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"]
Sep 17 18:52:13.406785 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:52:13.406761    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1f18a3_4851_4b0e_b1a0_c9399088a55b.slice/crio-2ea45907e0d6f8963cf5b844225ba7ade80721182d0dce873994adcf23c41e51 WatchSource:0}: Error finding container 2ea45907e0d6f8963cf5b844225ba7ade80721182d0dce873994adcf23c41e51: Status 404 returned error can't find the container with id 2ea45907e0d6f8963cf5b844225ba7ade80721182d0dce873994adcf23c41e51
Sep 17 18:52:13.937957 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.937918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerStarted","Data":"bc44e91d454f950c55fe88143730b5222f077f72dd7f6eb8d812defa14d226a2"}
Sep 17 18:52:13.940593 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:13.940515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerStarted","Data":"2ea45907e0d6f8963cf5b844225ba7ade80721182d0dce873994adcf23c41e51"}
Sep 17 18:52:17.969938 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:17.969898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerStarted","Data":"42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54"}
Sep 17 18:52:17.971506 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:17.971472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerStarted","Data":"ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb"}
Sep 17 18:52:21.987492 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:21.987457    2565 generic.go:356] "Generic (PLEG): container finished" podID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerID="42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54" exitCode=0
Sep 17 18:52:21.987935 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:21.987527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerDied","Data":"42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54"}
Sep 17 18:52:21.989423 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:21.989316    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerID="ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb" exitCode=0
Sep 17 18:52:21.989423 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:21.989377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerDied","Data":"ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb"}
Sep 17 18:52:38.095304 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:38.095232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerStarted","Data":"53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82"}
Sep 17 18:52:38.097143 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:38.097120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerStarted","Data":"9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b"}
Sep 17 18:52:41.112155 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:41.112119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerStarted","Data":"056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c"}
Sep 17 18:52:41.112645 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:41.112184    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:41.112645 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:41.112323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:41.113945 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:41.113909    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:52:41.114303 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:41.114287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerStarted","Data":"838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337"}
Sep 17 18:52:41.137482 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:41.137424    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podStartSLOduration=3.146179064 podStartE2EDuration="29.137410412s" totalImagesPullingTime="21.96019688s" totalInitContainerRuntime="4.031034468s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:41.135585952 +0000 UTC m=+850.421898469" watchObservedRunningTime="2026-09-17 18:52:41.137410412 +0000 UTC m=+850.423722991"
Sep 17 18:52:42.118739 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:42.118699    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:52:43.124285 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:43.124254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerStarted","Data":"0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43"}
Sep 17 18:52:43.124762 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:43.124529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:43.124762 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:43.124642    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:43.126102 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:43.126077    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:52:43.147357 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:43.147292    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podStartSLOduration=3.231134452 podStartE2EDuration="31.1472542s" totalImagesPullingTime="23.958913004s" totalInitContainerRuntime="3.957206744s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:12 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:43.14465963 +0000 UTC m=+852.430972159" watchObservedRunningTime="2026-09-17 18:52:43.1472542 +0000 UTC m=+852.433566717"
Sep 17 18:52:44.128278 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:44.128236    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:44.128742 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:44.128373    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:52:44.129573 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:44.129546    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:45.132453 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:45.132411    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:52:45.132899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:45.132846    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:45.136611 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:45.136591    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:52:46.135946 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:46.135903    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:52:46.136336 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:46.136152    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:47.122979 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:47.122952    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:52:47.123366 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:47.123343    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:52:56.135865 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:56.135821    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:52:56.136313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:56.136280    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:57.124411 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:52:57.124378    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:53:06.136624 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:06.136579    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:53:06.137038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:06.137010    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:07.123409 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:07.123370    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:53:16.136503 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:16.136399    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:53:16.136876 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:16.136846    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:17.124290 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:17.124256    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:53:26.136469 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:26.136408    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:53:26.136925 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:26.136904    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:27.124280 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:27.124239    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:53:36.135899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:36.135856    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:53:36.136304 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:36.136285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:37.123386 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:37.123348    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:53:46.136066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:46.136024    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:53:46.136607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:46.136583    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:47.123898 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:47.123862    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 18:53:54.900395 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:54.900362    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"]
Sep 17 18:53:54.900807 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:54.900582    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container" containerID="cri-o://53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82" gracePeriod=30
Sep 17 18:53:54.900807 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:54.900711    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kube-rbac-proxy" containerID="cri-o://056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c" gracePeriod=30
Sep 17 18:53:54.901621 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:54.901602    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:53:55.383143 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:55.383112    2565 generic.go:356] "Generic (PLEG): container finished" podID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerID="056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c" exitCode=2
Sep 17 18:53:55.383312 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:55.383185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerDied","Data":"056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c"}
Sep 17 18:53:56.136251 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:56.136222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:53:56.136643 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:56.136306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:53:57.119128 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:57.119083    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 18:53:58.834373 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.834350    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:53:58.893971 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.893946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df56d2a-69ae-4d50-b72f-260b00657bdf-kserve-provision-location\") pod \"2df56d2a-69ae-4d50-b72f-260b00657bdf\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") "
Sep 17 18:53:58.894081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.893979    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df56d2a-69ae-4d50-b72f-260b00657bdf-proxy-tls\") pod \"2df56d2a-69ae-4d50-b72f-260b00657bdf\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") "
Sep 17 18:53:58.894081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.894000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-cabundle-cert\") pod \"2df56d2a-69ae-4d50-b72f-260b00657bdf\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") "
Sep 17 18:53:58.894081 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.894044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pktr4\" (UniqueName: \"kubernetes.io/projected/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-api-access-pktr4\") pod \"2df56d2a-69ae-4d50-b72f-260b00657bdf\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") "
Sep 17 18:53:58.894203 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.894081    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-rbac-proxy-sar-config\") pod \"2df56d2a-69ae-4d50-b72f-260b00657bdf\" (UID: \"2df56d2a-69ae-4d50-b72f-260b00657bdf\") "
Sep 17 18:53:58.894298 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.894270    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df56d2a-69ae-4d50-b72f-260b00657bdf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2df56d2a-69ae-4d50-b72f-260b00657bdf" (UID: "2df56d2a-69ae-4d50-b72f-260b00657bdf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:58.894421 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.894306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2df56d2a-69ae-4d50-b72f-260b00657bdf" (UID: "2df56d2a-69ae-4d50-b72f-260b00657bdf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:58.894522 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.894499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2df56d2a-69ae-4d50-b72f-260b00657bdf" (UID: "2df56d2a-69ae-4d50-b72f-260b00657bdf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:58.896089 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.896064    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-api-access-pktr4" (OuterVolumeSpecName: "kube-api-access-pktr4") pod "2df56d2a-69ae-4d50-b72f-260b00657bdf" (UID: "2df56d2a-69ae-4d50-b72f-260b00657bdf"). InnerVolumeSpecName "kube-api-access-pktr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:58.896089 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.896075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df56d2a-69ae-4d50-b72f-260b00657bdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2df56d2a-69ae-4d50-b72f-260b00657bdf" (UID: "2df56d2a-69ae-4d50-b72f-260b00657bdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:58.995398 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.995375    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df56d2a-69ae-4d50-b72f-260b00657bdf-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:53:58.995398 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.995394    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df56d2a-69ae-4d50-b72f-260b00657bdf-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:53:58.995534 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.995422    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:53:58.995534 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.995431    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktr4\" (UniqueName: \"kubernetes.io/projected/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-api-access-pktr4\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:53:58.995534 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:58.995456    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df56d2a-69ae-4d50-b72f-260b00657bdf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:53:59.400024 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.399993    2565 generic.go:356] "Generic (PLEG): container finished" podID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerID="53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82" exitCode=0
Sep 17 18:53:59.400152 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.400059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"
Sep 17 18:53:59.400152 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.400075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerDied","Data":"53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82"}
Sep 17 18:53:59.400152 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.400133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc" event={"ID":"2df56d2a-69ae-4d50-b72f-260b00657bdf","Type":"ContainerDied","Data":"bc44e91d454f950c55fe88143730b5222f077f72dd7f6eb8d812defa14d226a2"}
Sep 17 18:53:59.400250 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.400161    2565 scope.go:117] "RemoveContainer" containerID="056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c"
Sep 17 18:53:59.407806 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.407791    2565 scope.go:117] "RemoveContainer" containerID="53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82"
Sep 17 18:53:59.414852 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.414836    2565 scope.go:117] "RemoveContainer" containerID="42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54"
Sep 17 18:53:59.421389 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.421374    2565 scope.go:117] "RemoveContainer" containerID="056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c"
Sep 17 18:53:59.421686 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:53:59.421668    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c\": container with ID starting with 056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c not found: ID does not exist" containerID="056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c"
Sep 17 18:53:59.421754 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.421694    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c"} err="failed to get container status \"056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c\": rpc error: code = NotFound desc = could not find container \"056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c\": container with ID starting with 056f4dc5f372d847a2ba16842b24416a6cdaf8ebe2e2e09dc0d9f15f608cff9c not found: ID does not exist"
Sep 17 18:53:59.421754 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.421709    2565 scope.go:117] "RemoveContainer" containerID="53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82"
Sep 17 18:53:59.421962 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:53:59.421946    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82\": container with ID starting with 53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82 not found: ID does not exist" containerID="53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82"
Sep 17 18:53:59.422008 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.421967    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82"} err="failed to get container status \"53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82\": rpc error: code = NotFound desc = could not find container \"53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82\": container with ID starting with 53856dff16d944ce6b78e6cc457dfb85abea1b9cb2375a1197947b7a66fcfb82 not found: ID does not exist"
Sep 17 18:53:59.422008 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.421981    2565 scope.go:117] "RemoveContainer" containerID="42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54"
Sep 17 18:53:59.422211 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:53:59.422194    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54\": container with ID starting with 42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54 not found: ID does not exist" containerID="42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54"
Sep 17 18:53:59.422261 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.422219    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54"} err="failed to get container status \"42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54\": rpc error: code = NotFound desc = could not find container \"42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54\": container with ID starting with 42e37623ec8d73aed9e24dc81a4d250b61beb3d11031e4315d515a54650a2f54 not found: ID does not exist"
Sep 17 18:53:59.426586 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.426565    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"]
Sep 17 18:53:59.432011 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:53:59.431992    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-gd8xc"]
Sep 17 18:54:01.061916 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:01.061886    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" path="/var/lib/kubelet/pods/2df56d2a-69ae-4d50-b72f-260b00657bdf/volumes"
Sep 17 18:54:02.647500 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647465    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"]
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647744    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kube-rbac-proxy"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647755    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kube-rbac-proxy"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647763    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647769    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647787    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="storage-initializer"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="storage-initializer"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647840    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kserve-container"
Sep 17 18:54:02.647862 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.647846    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df56d2a-69ae-4d50-b72f-260b00657bdf" containerName="kube-rbac-proxy"
Sep 17 18:54:02.652273 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.652255    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.655810 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.655789    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:54:02.656017 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.656000    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:54:02.674529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.674509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"]
Sep 17 18:54:02.717237 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.717213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.717354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.717244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d8def-6d0d-4090-94ed-22a729b5683d-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.717354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.717265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.717354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.717287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791d8def-6d0d-4090-94ed-22a729b5683d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.717354 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.717336    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7br28\" (UniqueName: \"kubernetes.io/projected/791d8def-6d0d-4090-94ed-22a729b5683d-kube-api-access-7br28\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.818586 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.818554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791d8def-6d0d-4090-94ed-22a729b5683d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.818719 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.818593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7br28\" (UniqueName: \"kubernetes.io/projected/791d8def-6d0d-4090-94ed-22a729b5683d-kube-api-access-7br28\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.818719 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.818633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.818719 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.818656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d8def-6d0d-4090-94ed-22a729b5683d-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.818719 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.818677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.819071 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.819047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791d8def-6d0d-4090-94ed-22a729b5683d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.819321 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.819295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.819426 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.819333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.821032 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.821013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d8def-6d0d-4090-94ed-22a729b5683d-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.827500 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.827481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7br28\" (UniqueName: \"kubernetes.io/projected/791d8def-6d0d-4090-94ed-22a729b5683d-kube-api-access-7br28\") pod \"isvc-canary-rollback-predictor-69978998f6-8mjrd\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:02.961270 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:02.961248    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:03.232658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:03.232635    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"]
Sep 17 18:54:03.234942 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:54:03.234904    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791d8def_6d0d_4090_94ed_22a729b5683d.slice/crio-61223b429067315396218397f4e2afb655bb16ef1230dd55569423a9cb47768b WatchSource:0}: Error finding container 61223b429067315396218397f4e2afb655bb16ef1230dd55569423a9cb47768b: Status 404 returned error can't find the container with id 61223b429067315396218397f4e2afb655bb16ef1230dd55569423a9cb47768b
Sep 17 18:54:03.419006 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:03.418974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerStarted","Data":"ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048"}
Sep 17 18:54:03.419155 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:03.419032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerStarted","Data":"61223b429067315396218397f4e2afb655bb16ef1230dd55569423a9cb47768b"}
Sep 17 18:54:07.433913 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:07.433884    2565 generic.go:356] "Generic (PLEG): container finished" podID="791d8def-6d0d-4090-94ed-22a729b5683d" containerID="ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048" exitCode=0
Sep 17 18:54:07.434199 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:07.433938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerDied","Data":"ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048"}
Sep 17 18:54:07.657418 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:07.657331    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"]
Sep 17 18:54:07.657649 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:07.657619    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" containerID="cri-o://9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b" gracePeriod=30
Sep 17 18:54:07.657793 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:07.657635    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" containerID="cri-o://0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43" gracePeriod=30
Sep 17 18:54:07.657793 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:07.657674    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" containerID="cri-o://838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337" gracePeriod=30
Sep 17 18:54:08.440537 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.440493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerStarted","Data":"a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378"}
Sep 17 18:54:08.440997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.440567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerStarted","Data":"713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4"}
Sep 17 18:54:08.440997 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.440834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:08.441131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.441007    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:08.442567 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.442537    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:54:08.443277 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.443255    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerID="838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337" exitCode=2
Sep 17 18:54:08.443386 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.443314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerDied","Data":"838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337"}
Sep 17 18:54:08.469880 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:08.469839    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podStartSLOduration=2.427806747 podStartE2EDuration="6.469820521s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.042013774s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:08.469241934 +0000 UTC m=+937.755554450" watchObservedRunningTime="2026-09-17 18:54:08.469820521 +0000 UTC m=+937.756133036"
Sep 17 18:54:09.448127 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:09.448083    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:54:10.133037 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:10.132996    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 18:54:12.461582 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:12.461551    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerID="9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b" exitCode=0
Sep 17 18:54:12.461941 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:12.461623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerDied","Data":"9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b"}
Sep 17 18:54:14.452745 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:14.452717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:54:14.453203 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:14.453176    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:54:15.133494 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:15.133459    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 18:54:16.136241 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:16.136194    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:54:16.136680 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:16.136558    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:20.132750 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:20.132708    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 18:54:20.135039 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:20.132841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:54:24.453978 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:24.453942    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:54:25.132974 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:25.132934    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 18:54:26.136625 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:26.136583    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:54:26.137032 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:26.136915    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:30.133378 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:30.133340    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 18:54:34.453793 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:34.453756    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:54:35.133155 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:35.133119    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 18:54:36.136499 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:36.136456    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 18:54:36.136901 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:36.136593    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:54:36.136901 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:36.136826    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:36.136995 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:36.136947    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:54:37.795907 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.795884    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:54:37.864589 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.864559    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkr8m\" (UniqueName: \"kubernetes.io/projected/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-api-access-nkr8m\") pod \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") "
Sep 17 18:54:37.864797 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.864618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-cabundle-cert\") pod \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") "
Sep 17 18:54:37.864797 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.864677    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-proxy-tls\") pod \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") "
Sep 17 18:54:37.864797 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.864697    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kserve-provision-location\") pod \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") "
Sep 17 18:54:37.864994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.864834    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-rbac-proxy-sar-config\") pod \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\" (UID: \"7d1f18a3-4851-4b0e-b1a0-c9399088a55b\") "
Sep 17 18:54:37.865082 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.865059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d1f18a3-4851-4b0e-b1a0-c9399088a55b" (UID: "7d1f18a3-4851-4b0e-b1a0-c9399088a55b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:37.865133 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.865060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d1f18a3-4851-4b0e-b1a0-c9399088a55b" (UID: "7d1f18a3-4851-4b0e-b1a0-c9399088a55b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:37.865188 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.865144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d1f18a3-4851-4b0e-b1a0-c9399088a55b" (UID: "7d1f18a3-4851-4b0e-b1a0-c9399088a55b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:37.865188 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.865150    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:37.865188 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.865173    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:37.866699 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.866676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-api-access-nkr8m" (OuterVolumeSpecName: "kube-api-access-nkr8m") pod "7d1f18a3-4851-4b0e-b1a0-c9399088a55b" (UID: "7d1f18a3-4851-4b0e-b1a0-c9399088a55b"). InnerVolumeSpecName "kube-api-access-nkr8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:37.866699 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.866680    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d1f18a3-4851-4b0e-b1a0-c9399088a55b" (UID: "7d1f18a3-4851-4b0e-b1a0-c9399088a55b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:37.965592 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.965568    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkr8m\" (UniqueName: \"kubernetes.io/projected/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-api-access-nkr8m\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:37.965592 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.965588    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:37.965725 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:37.965598    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d1f18a3-4851-4b0e-b1a0-c9399088a55b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:38.555423 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.555390    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerID="0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43" exitCode=0
Sep 17 18:54:38.555589 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.555479    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"
Sep 17 18:54:38.555589 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.555478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerDied","Data":"0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43"}
Sep 17 18:54:38.555589 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.555556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57" event={"ID":"7d1f18a3-4851-4b0e-b1a0-c9399088a55b","Type":"ContainerDied","Data":"2ea45907e0d6f8963cf5b844225ba7ade80721182d0dce873994adcf23c41e51"}
Sep 17 18:54:38.555683 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.555604    2565 scope.go:117] "RemoveContainer" containerID="0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43"
Sep 17 18:54:38.565946 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.565922    2565 scope.go:117] "RemoveContainer" containerID="838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337"
Sep 17 18:54:38.573993 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.573977    2565 scope.go:117] "RemoveContainer" containerID="9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b"
Sep 17 18:54:38.580595 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.580576    2565 scope.go:117] "RemoveContainer" containerID="ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb"
Sep 17 18:54:38.584372 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.584352    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"]
Sep 17 18:54:38.587890 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.587869    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fe81c-predictor-c59f58b57-k9z57"]
Sep 17 18:54:38.589043 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589028    2565 scope.go:117] "RemoveContainer" containerID="0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43"
Sep 17 18:54:38.589336 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:54:38.589310    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43\": container with ID starting with 0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43 not found: ID does not exist" containerID="0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43"
Sep 17 18:54:38.589423 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589339    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43"} err="failed to get container status \"0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43\": rpc error: code = NotFound desc = could not find container \"0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43\": container with ID starting with 0c0af240d499d17b102959f63015d454e175deb1824b8ff5c66d4b9e442a5e43 not found: ID does not exist"
Sep 17 18:54:38.589423 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589356    2565 scope.go:117] "RemoveContainer" containerID="838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337"
Sep 17 18:54:38.589648 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:54:38.589629    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337\": container with ID starting with 838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337 not found: ID does not exist" containerID="838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337"
Sep 17 18:54:38.589717 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589659    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337"} err="failed to get container status \"838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337\": rpc error: code = NotFound desc = could not find container \"838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337\": container with ID starting with 838fa2b65713bb7489b93f0385b0cfbfaff8000380835300aeade184d572f337 not found: ID does not exist"
Sep 17 18:54:38.589717 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589682    2565 scope.go:117] "RemoveContainer" containerID="9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b"
Sep 17 18:54:38.589959 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:54:38.589918    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b\": container with ID starting with 9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b not found: ID does not exist" containerID="9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b"
Sep 17 18:54:38.589959 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589944    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b"} err="failed to get container status \"9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b\": rpc error: code = NotFound desc = could not find container \"9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b\": container with ID starting with 9e06772c9c3deb262ab502b5b27dd318d05697cf617c5c4dfb4521ab9850390b not found: ID does not exist"
Sep 17 18:54:38.589959 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.589959    2565 scope.go:117] "RemoveContainer" containerID="ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb"
Sep 17 18:54:38.590224 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:54:38.590210    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb\": container with ID starting with ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb not found: ID does not exist" containerID="ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb"
Sep 17 18:54:38.590267 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:38.590226    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb"} err="failed to get container status \"ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb\": rpc error: code = NotFound desc = could not find container \"ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb\": container with ID starting with ab88524608f6f4a0e23c81c2d41788dde94990704edd4eeb0cb4fd7f6a186abb not found: ID does not exist"
Sep 17 18:54:39.061740 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:39.061703    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" path="/var/lib/kubelet/pods/7d1f18a3-4851-4b0e-b1a0-c9399088a55b/volumes"
Sep 17 18:54:44.454083 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:44.454046    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:54:54.453115 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:54:54.453078    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:55:04.453883 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:04.453847    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:55:14.453431 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:14.453390    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:55:24.454599 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:24.454569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:55:32.641043 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:32.641000    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"]
Sep 17 18:55:32.641550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:32.641227    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" containerID="cri-o://713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4" gracePeriod=30
Sep 17 18:55:32.641550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:32.641263    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kube-rbac-proxy" containerID="cri-o://a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378" gracePeriod=30
Sep 17 18:55:33.757233 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:33.757201    2565 generic.go:356] "Generic (PLEG): container finished" podID="791d8def-6d0d-4090-94ed-22a729b5683d" containerID="a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378" exitCode=2
Sep 17 18:55:33.757593 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:33.757274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerDied","Data":"a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378"}
Sep 17 18:55:34.449082 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:34.449047    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 17 18:55:34.453388 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:34.453362    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:55:36.674406 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.674384    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:55:36.768746 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.768717    2565 generic.go:356] "Generic (PLEG): container finished" podID="791d8def-6d0d-4090-94ed-22a729b5683d" containerID="713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4" exitCode=0
Sep 17 18:55:36.768872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.768765    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"
Sep 17 18:55:36.768872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.768792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerDied","Data":"713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4"}
Sep 17 18:55:36.768872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.768841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd" event={"ID":"791d8def-6d0d-4090-94ed-22a729b5683d","Type":"ContainerDied","Data":"61223b429067315396218397f4e2afb655bb16ef1230dd55569423a9cb47768b"}
Sep 17 18:55:36.768872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.768870    2565 scope.go:117] "RemoveContainer" containerID="a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378"
Sep 17 18:55:36.772057 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791d8def-6d0d-4090-94ed-22a729b5683d-kserve-provision-location\") pod \"791d8def-6d0d-4090-94ed-22a729b5683d\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") "
Sep 17 18:55:36.772192 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7br28\" (UniqueName: \"kubernetes.io/projected/791d8def-6d0d-4090-94ed-22a729b5683d-kube-api-access-7br28\") pod \"791d8def-6d0d-4090-94ed-22a729b5683d\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") "
Sep 17 18:55:36.772192 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-cabundle-cert\") pod \"791d8def-6d0d-4090-94ed-22a729b5683d\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") "
Sep 17 18:55:36.772192 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-kube-rbac-proxy-sar-config\") pod \"791d8def-6d0d-4090-94ed-22a729b5683d\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") "
Sep 17 18:55:36.772192 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d8def-6d0d-4090-94ed-22a729b5683d-proxy-tls\") pod \"791d8def-6d0d-4090-94ed-22a729b5683d\" (UID: \"791d8def-6d0d-4090-94ed-22a729b5683d\") "
Sep 17 18:55:36.772452 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791d8def-6d0d-4090-94ed-22a729b5683d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "791d8def-6d0d-4090-94ed-22a729b5683d" (UID: "791d8def-6d0d-4090-94ed-22a729b5683d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:36.772561 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "791d8def-6d0d-4090-94ed-22a729b5683d" (UID: "791d8def-6d0d-4090-94ed-22a729b5683d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:55:36.772635 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.772602    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "791d8def-6d0d-4090-94ed-22a729b5683d" (UID: "791d8def-6d0d-4090-94ed-22a729b5683d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:55:36.774283 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.774257    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791d8def-6d0d-4090-94ed-22a729b5683d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "791d8def-6d0d-4090-94ed-22a729b5683d" (UID: "791d8def-6d0d-4090-94ed-22a729b5683d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:55:36.774391 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.774321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d8def-6d0d-4090-94ed-22a729b5683d-kube-api-access-7br28" (OuterVolumeSpecName: "kube-api-access-7br28") pod "791d8def-6d0d-4090-94ed-22a729b5683d" (UID: "791d8def-6d0d-4090-94ed-22a729b5683d"). InnerVolumeSpecName "kube-api-access-7br28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:55:36.777734 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.777717    2565 scope.go:117] "RemoveContainer" containerID="713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4"
Sep 17 18:55:36.786912 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.786895    2565 scope.go:117] "RemoveContainer" containerID="ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048"
Sep 17 18:55:36.793405 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.793389    2565 scope.go:117] "RemoveContainer" containerID="a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378"
Sep 17 18:55:36.793681 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:36.793664    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378\": container with ID starting with a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378 not found: ID does not exist" containerID="a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378"
Sep 17 18:55:36.793741 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.793689    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378"} err="failed to get container status \"a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378\": rpc error: code = NotFound desc = could not find container \"a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378\": container with ID starting with a0b60f1fc1095f01c0af719fae10068415b05715754ff7bf17b7ac1f09c9f378 not found: ID does not exist"
Sep 17 18:55:36.793741 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.793704    2565 scope.go:117] "RemoveContainer" containerID="713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4"
Sep 17 18:55:36.793960 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:36.793942    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4\": container with ID starting with 713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4 not found: ID does not exist" containerID="713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4"
Sep 17 18:55:36.793999 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.793965    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4"} err="failed to get container status \"713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4\": rpc error: code = NotFound desc = could not find container \"713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4\": container with ID starting with 713f614ffed674ed5d057039290a58b3cf37a6b60152c0899b2850ccac39dcb4 not found: ID does not exist"
Sep 17 18:55:36.793999 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.793981    2565 scope.go:117] "RemoveContainer" containerID="ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048"
Sep 17 18:55:36.794232 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:36.794218    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048\": container with ID starting with ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048 not found: ID does not exist" containerID="ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048"
Sep 17 18:55:36.794274 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.794237    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048"} err="failed to get container status \"ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048\": rpc error: code = NotFound desc = could not find container \"ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048\": container with ID starting with ab81281ae5c4159cb0a808063839a74325794da2a7e7091bc02710f05957a048 not found: ID does not exist"
Sep 17 18:55:36.873535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.873514    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d8def-6d0d-4090-94ed-22a729b5683d-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:55:36.873535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.873532    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791d8def-6d0d-4090-94ed-22a729b5683d-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:55:36.873669 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.873543    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7br28\" (UniqueName: \"kubernetes.io/projected/791d8def-6d0d-4090-94ed-22a729b5683d-kube-api-access-7br28\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:55:36.873669 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.873551    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:55:36.873669 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:36.873559    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d8def-6d0d-4090-94ed-22a729b5683d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:55:37.084964 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.084936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"]
Sep 17 18:55:37.089334 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.089315    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-8mjrd"]
Sep 17 18:55:37.899500 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899472    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"]
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899723    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899733    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899744    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899749    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899758    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="storage-initializer"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899763    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="storage-initializer"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899771    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899775    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899782    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899787    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899793    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kube-rbac-proxy"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kube-rbac-proxy"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899808    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="storage-initializer"
Sep 17 18:55:37.899841 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899815    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="storage-initializer"
Sep 17 18:55:37.900258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899857    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kube-rbac-proxy"
Sep 17 18:55:37.900258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899865    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="agent"
Sep 17 18:55:37.900258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899871    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kube-rbac-proxy"
Sep 17 18:55:37.900258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899876    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1f18a3-4851-4b0e-b1a0-c9399088a55b" containerName="kserve-container"
Sep 17 18:55:37.900258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.899882    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" containerName="kserve-container"
Sep 17 18:55:37.904268 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.904250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:37.910140 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.909961    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:55:37.910140 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.910013    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:37.913364 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.913347    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:55:37.913473 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.913375    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:55:37.913634 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.913619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 18:55:37.913982 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.913967    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:55:37.914059 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.914037    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:55:37.925835 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.925806    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"]
Sep 17 18:55:37.973224 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.973194    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"]
Sep 17 18:55:37.976429 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.976411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:37.980351 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.980336    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:55:37.992300 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:37.992279    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"]
Sep 17 18:55:38.082981 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.082956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.083094 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.082985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.083094 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fssr\" (UniqueName: \"kubernetes.io/projected/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-api-access-4fssr\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.083094 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f2258-8e06-4e3a-8add-e39a383ece3c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.083094 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.083094 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1efd46be-4254-4685-8025-88b44bfd8380-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.083253 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.083253 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1efd46be-4254-4685-8025-88b44bfd8380-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.083253 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.083253 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.083204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgmfb\" (UniqueName: \"kubernetes.io/projected/1efd46be-4254-4685-8025-88b44bfd8380-kube-api-access-vgmfb\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.183931 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.183906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f2258-8e06-4e3a-8add-e39a383ece3c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.183941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.183959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1efd46be-4254-4685-8025-88b44bfd8380-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.183975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.183999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1efd46be-4254-4685-8025-88b44bfd8380-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184066 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgmfb\" (UniqueName: \"kubernetes.io/projected/1efd46be-4254-4685-8025-88b44bfd8380-kube-api-access-vgmfb\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184335 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184335 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184335 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fssr\" (UniqueName: \"kubernetes.io/projected/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-api-access-4fssr\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184503 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1efd46be-4254-4685-8025-88b44bfd8380-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184503 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:38.184361    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:55:38.184503 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:38.184461    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls podName:e00f2258-8e06-4e3a-8add-e39a383ece3c nodeName:}" failed. No retries permitted until 2026-09-17 18:55:38.684413996 +0000 UTC m=+1027.970726507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls") pod "isvc-canary-stop-predictor-6958d6c864-r74ps" (UID: "e00f2258-8e06-4e3a-8add-e39a383ece3c") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:55:38.184503 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f2258-8e06-4e3a-8add-e39a383ece3c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184763 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184801 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.184832 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.184874 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.184859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.186814 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.186796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1efd46be-4254-4685-8025-88b44bfd8380-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.193713 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.193693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fssr\" (UniqueName: \"kubernetes.io/projected/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-api-access-4fssr\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.195281 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.195261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgmfb\" (UniqueName: \"kubernetes.io/projected/1efd46be-4254-4685-8025-88b44bfd8380-kube-api-access-vgmfb\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-fkdgw\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.213108 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.213086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:38.489288 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.489217    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"]
Sep 17 18:55:38.492411 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:55:38.492385    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efd46be_4254_4685_8025_88b44bfd8380.slice/crio-a76c62aa753af56120054bae0d9bc6a425046bb4b25da47a7bbcf92af8852937 WatchSource:0}: Error finding container a76c62aa753af56120054bae0d9bc6a425046bb4b25da47a7bbcf92af8852937: Status 404 returned error can't find the container with id a76c62aa753af56120054bae0d9bc6a425046bb4b25da47a7bbcf92af8852937
Sep 17 18:55:38.687966 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.687938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.690085 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.690067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-r74ps\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:38.778579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.778498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerStarted","Data":"d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5"}
Sep 17 18:55:38.778579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.778559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerStarted","Data":"a76c62aa753af56120054bae0d9bc6a425046bb4b25da47a7bbcf92af8852937"}
Sep 17 18:55:38.885557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:38.885528    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:39.061796 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:39.061721    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791d8def-6d0d-4090-94ed-22a729b5683d" path="/var/lib/kubelet/pods/791d8def-6d0d-4090-94ed-22a729b5683d/volumes"
Sep 17 18:55:39.162265 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:39.162235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"]
Sep 17 18:55:39.165498 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:55:39.165477    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00f2258_8e06_4e3a_8add_e39a383ece3c.slice/crio-a77861b13a4a483f373b06d69e08c18b3a5db357f58278a9f6f259cccd590944 WatchSource:0}: Error finding container a77861b13a4a483f373b06d69e08c18b3a5db357f58278a9f6f259cccd590944: Status 404 returned error can't find the container with id a77861b13a4a483f373b06d69e08c18b3a5db357f58278a9f6f259cccd590944
Sep 17 18:55:39.784247 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:39.784205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerStarted","Data":"176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020"}
Sep 17 18:55:39.784455 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:39.784289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerStarted","Data":"a77861b13a4a483f373b06d69e08c18b3a5db357f58278a9f6f259cccd590944"}
Sep 17 18:55:42.797093 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:42.797063    2565 generic.go:356] "Generic (PLEG): container finished" podID="1efd46be-4254-4685-8025-88b44bfd8380" containerID="d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5" exitCode=0
Sep 17 18:55:42.797450 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:42.797107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerDied","Data":"d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5"}
Sep 17 18:55:43.803636 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:43.803604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerStarted","Data":"452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d"}
Sep 17 18:55:43.804036 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:43.803664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerStarted","Data":"10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112"}
Sep 17 18:55:43.804036 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:43.803840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:43.805239 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:43.805217    2565 generic.go:356] "Generic (PLEG): container finished" podID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerID="176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020" exitCode=0
Sep 17 18:55:43.805319 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:43.805254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerDied","Data":"176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020"}
Sep 17 18:55:43.827424 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:43.827386    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podStartSLOduration=3.10300461 podStartE2EDuration="6.82737248s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.72436787s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:43.825472919 +0000 UTC m=+1033.111785432" watchObservedRunningTime="2026-09-17 18:55:43.82737248 +0000 UTC m=+1033.113684995"
Sep 17 18:55:44.811452 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.811396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerStarted","Data":"ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195"}
Sep 17 18:55:44.811780 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.811709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:44.811780 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.811765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerStarted","Data":"a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4"}
Sep 17 18:55:44.812056 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.812031    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:44.812184 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.812167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:44.813093 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.813072    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:55:44.813297 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.813274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:55:44.834379 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:44.834336    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podStartSLOduration=3.972801443 podStartE2EDuration="7.834323739s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.861522296s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:44.833305053 +0000 UTC m=+1034.119617572" watchObservedRunningTime="2026-09-17 18:55:44.834323739 +0000 UTC m=+1034.120636255"
Sep 17 18:55:45.815708 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:45.815665    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:55:45.816104 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:45.815866    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:55:46.819554 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:46.819515    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:55:47.866342 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.866313    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"]
Sep 17 18:55:47.869757 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.869743    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:47.873041 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.873014    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-bae88-serving-cert\""
Sep 17 18:55:47.873851 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.873835    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-bae88-kube-rbac-proxy-sar-config\""
Sep 17 18:55:47.881271 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.881251    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"]
Sep 17 18:55:47.954793 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.954770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:47.954897 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:47.954806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e646460f-7720-4476-ae7c-836d695635bc-openshift-service-ca-bundle\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:48.055250 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:48.055228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:48.055358 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:48.055268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e646460f-7720-4476-ae7c-836d695635bc-openshift-service-ca-bundle\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:48.055416 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:48.055365    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-bae88-serving-cert: secret "model-chainer-raw-bae88-serving-cert" not found
Sep 17 18:55:48.055496 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:55:48.055426    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls podName:e646460f-7720-4476-ae7c-836d695635bc nodeName:}" failed. No retries permitted until 2026-09-17 18:55:48.555412491 +0000 UTC m=+1037.841724991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls") pod "model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" (UID: "e646460f-7720-4476-ae7c-836d695635bc") : secret "model-chainer-raw-bae88-serving-cert" not found
Sep 17 18:55:48.055904 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:48.055883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e646460f-7720-4476-ae7c-836d695635bc-openshift-service-ca-bundle\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:48.560542 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:48.560508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:48.562900 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:48.562870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls\") pod \"model-chainer-raw-bae88-7f9cfbdbc5-wpkqf\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:48.778657 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:48.778613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:49.053863 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:49.053828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"]
Sep 17 18:55:49.056451 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:55:49.056413    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode646460f_7720_4476_ae7c_836d695635bc.slice/crio-d362bf462b3516dbce8f88b284ca9a9c18e81a396eb0773e3daa6522f52b45ff WatchSource:0}: Error finding container d362bf462b3516dbce8f88b284ca9a9c18e81a396eb0773e3daa6522f52b45ff: Status 404 returned error can't find the container with id d362bf462b3516dbce8f88b284ca9a9c18e81a396eb0773e3daa6522f52b45ff
Sep 17 18:55:49.836053 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:49.836019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" event={"ID":"e646460f-7720-4476-ae7c-836d695635bc","Type":"ContainerStarted","Data":"d362bf462b3516dbce8f88b284ca9a9c18e81a396eb0773e3daa6522f52b45ff"}
Sep 17 18:55:50.820642 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:50.820610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:55:50.821277 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:50.821243    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:55:51.824371 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:51.824346    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:55:51.824807 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:51.824781    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:55:51.847027 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:51.847000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" event={"ID":"e646460f-7720-4476-ae7c-836d695635bc","Type":"ContainerStarted","Data":"07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497"}
Sep 17 18:55:51.847164 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:51.847112    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:51.880744 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:51.880703    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podStartSLOduration=2.651054602 podStartE2EDuration="4.880688564s" totalImagesPullingTime="2.229633962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:51.877930578 +0000 UTC m=+1041.164243094" watchObservedRunningTime="2026-09-17 18:55:51.880688564 +0000 UTC m=+1041.167001080"
Sep 17 18:55:57.857038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:57.857008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:55:57.887120 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:57.887090    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"]
Sep 17 18:55:57.887306 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:57.887283    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" containerID="cri-o://07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497" gracePeriod=30
Sep 17 18:55:58.416870 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.416836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"]
Sep 17 18:55:58.420349 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.420334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.422890 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.422870    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-serving-cert\""
Sep 17 18:55:58.423313 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.423299    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-8b7b7-kube-rbac-proxy-sar-config\""
Sep 17 18:55:58.431791 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.431769    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"]
Sep 17 18:55:58.533143 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.533120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2d5a63-885b-406b-bddc-480cdb1a73c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.533258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.533154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.533258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.533173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.533258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.533190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcjx\" (UniqueName: \"kubernetes.io/projected/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-api-access-dxcjx\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.533258 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.533236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634028 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.633999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2d5a63-885b-406b-bddc-480cdb1a73c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634150 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634150 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634150 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcjx\" (UniqueName: \"kubernetes.io/projected/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-api-access-dxcjx\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634150 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634726 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.634817 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.634801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.636508 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.636490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2d5a63-885b-406b-bddc-480cdb1a73c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.658446 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.658406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcjx\" (UniqueName: \"kubernetes.io/projected/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-api-access-dxcjx\") pod \"isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:58.730431 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:58.730409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:55:59.005550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:59.005527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"]
Sep 17 18:55:59.007425 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:55:59.007396    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2d5a63_885b_406b_bddc_480cdb1a73c0.slice/crio-e254ab2590f17ed8b8b0b07549caee1a867df81d9233b31dce5e94f8a252b177 WatchSource:0}: Error finding container e254ab2590f17ed8b8b0b07549caee1a867df81d9233b31dce5e94f8a252b177: Status 404 returned error can't find the container with id e254ab2590f17ed8b8b0b07549caee1a867df81d9233b31dce5e94f8a252b177
Sep 17 18:55:59.878948 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:59.878907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerStarted","Data":"0cf13c014e1a31601acc9847848f556e43f95e09d4b157e860ff88bb19c20f27"}
Sep 17 18:55:59.879089 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:55:59.878984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerStarted","Data":"e254ab2590f17ed8b8b0b07549caee1a867df81d9233b31dce5e94f8a252b177"}
Sep 17 18:56:00.821627 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:00.821588    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:01.825351 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:01.825311    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:02.855934 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:02.855896    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:02.892780 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:02.892755    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerID="0cf13c014e1a31601acc9847848f556e43f95e09d4b157e860ff88bb19c20f27" exitCode=0
Sep 17 18:56:02.892916 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:02.892831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerDied","Data":"0cf13c014e1a31601acc9847848f556e43f95e09d4b157e860ff88bb19c20f27"}
Sep 17 18:56:07.855803 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:07.855769    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:10.821700 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:10.821662    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:11.824916 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:11.824882    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:12.855639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:12.855602    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:12.855992 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:12.855701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:56:17.855792 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:17.855712    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:20.824366 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:20.824316    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:21.825514 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:21.825477    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:22.855283 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:22.855244    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:22.981063 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:22.981029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerStarted","Data":"415f345c87f01ff204f96bd0bff6df4e3d1c2c7cdef7f05acf8a65ddbb9a774e"}
Sep 17 18:56:22.981208 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:22.981091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerStarted","Data":"b51fc2bd7c38d1815453e9a210a982818260f7c28f5778a2eecb0b5a590391be"}
Sep 17 18:56:22.981315 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:22.981297    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:56:23.001886 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:23.001835    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podStartSLOduration=2.105182933 podStartE2EDuration="25.001822585s" totalImagesPullingTime="19.261731944s" totalInitContainerRuntime="3.634907708s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:23.000532319 +0000 UTC m=+1072.286844864" watchObservedRunningTime="2026-09-17 18:56:23.001822585 +0000 UTC m=+1072.288135100"
Sep 17 18:56:23.985118 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:23.985090    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:56:23.986391 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:23.986363    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:56:24.989129 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:24.989085    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:56:27.856166 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:27.856127    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:28.586077 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.586049    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:56:28.691208 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.691179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e646460f-7720-4476-ae7c-836d695635bc-openshift-service-ca-bundle\") pod \"e646460f-7720-4476-ae7c-836d695635bc\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") "
Sep 17 18:56:28.691208 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.691212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls\") pod \"e646460f-7720-4476-ae7c-836d695635bc\" (UID: \"e646460f-7720-4476-ae7c-836d695635bc\") "
Sep 17 18:56:28.691554 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.691525    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e646460f-7720-4476-ae7c-836d695635bc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e646460f-7720-4476-ae7c-836d695635bc" (UID: "e646460f-7720-4476-ae7c-836d695635bc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:28.693168 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.693143    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e646460f-7720-4476-ae7c-836d695635bc" (UID: "e646460f-7720-4476-ae7c-836d695635bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:28.791786 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.791721    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e646460f-7720-4476-ae7c-836d695635bc-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:56:28.791786 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:28.791748    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e646460f-7720-4476-ae7c-836d695635bc-openshift-service-ca-bundle\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.005207 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.005172    2565 generic.go:356] "Generic (PLEG): container finished" podID="e646460f-7720-4476-ae7c-836d695635bc" containerID="07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497" exitCode=0
Sep 17 18:56:29.005640 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.005213    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"
Sep 17 18:56:29.005640 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.005258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" event={"ID":"e646460f-7720-4476-ae7c-836d695635bc","Type":"ContainerDied","Data":"07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497"}
Sep 17 18:56:29.005640 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.005308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf" event={"ID":"e646460f-7720-4476-ae7c-836d695635bc","Type":"ContainerDied","Data":"d362bf462b3516dbce8f88b284ca9a9c18e81a396eb0773e3daa6522f52b45ff"}
Sep 17 18:56:29.005640 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.005332    2565 scope.go:117] "RemoveContainer" containerID="07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497"
Sep 17 18:56:29.013909 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.013892    2565 scope.go:117] "RemoveContainer" containerID="07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497"
Sep 17 18:56:29.014161 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:56:29.014142    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497\": container with ID starting with 07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497 not found: ID does not exist" containerID="07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497"
Sep 17 18:56:29.014228 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.014168    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497"} err="failed to get container status \"07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497\": rpc error: code = NotFound desc = could not find container \"07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497\": container with ID starting with 07008537b01d5a701204f8ed058d9b736ed45158e64b20876636395aaa1a8497 not found: ID does not exist"
Sep 17 18:56:29.028077 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.028050    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"]
Sep 17 18:56:29.032821 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.032802    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bae88-7f9cfbdbc5-wpkqf"]
Sep 17 18:56:29.061718 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.061657    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e646460f-7720-4476-ae7c-836d695635bc" path="/var/lib/kubelet/pods/e646460f-7720-4476-ae7c-836d695635bc/volumes"
Sep 17 18:56:29.993994 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.993967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:56:29.994593 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:29.994564    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:56:30.822490 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:30.822431    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:31.825734 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:31.825700    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:39.995049 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:39.995008    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:56:40.821065 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:40.821025    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:41.824931 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:41.824886    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:49.995177 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:49.995134    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:56:50.822001 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:50.821962    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:51.825111 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:51.825067    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:53.058790 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:53.058756    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:56:53.059190 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:53.058756    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:56:59.994869 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:56:59.994827    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:57:03.063024 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:03.062996    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:57:03.063382 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:03.063068    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:57:03.243418 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:03.243385    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"]
Sep 17 18:57:03.243651 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:03.243628    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container" containerID="cri-o://a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4" gracePeriod=30
Sep 17 18:57:03.243729 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:03.243651    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kube-rbac-proxy" containerID="cri-o://ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195" gracePeriod=30
Sep 17 18:57:04.150703 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:04.150673    2565 generic.go:356] "Generic (PLEG): container finished" podID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerID="ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195" exitCode=2
Sep 17 18:57:04.151057 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:04.150740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerDied","Data":"ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195"}
Sep 17 18:57:06.820488 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:06.820433    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 18:57:07.390167 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.390137    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:57:07.469764 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.469735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fssr\" (UniqueName: \"kubernetes.io/projected/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-api-access-4fssr\") pod \"e00f2258-8e06-4e3a-8add-e39a383ece3c\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") "
Sep 17 18:57:07.469932 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.469782    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f2258-8e06-4e3a-8add-e39a383ece3c-kserve-provision-location\") pod \"e00f2258-8e06-4e3a-8add-e39a383ece3c\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") "
Sep 17 18:57:07.469932 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.469811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-cabundle-cert\") pod \"e00f2258-8e06-4e3a-8add-e39a383ece3c\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") "
Sep 17 18:57:07.469932 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.469866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-rbac-proxy-sar-config\") pod \"e00f2258-8e06-4e3a-8add-e39a383ece3c\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") "
Sep 17 18:57:07.469932 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.469887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls\") pod \"e00f2258-8e06-4e3a-8add-e39a383ece3c\" (UID: \"e00f2258-8e06-4e3a-8add-e39a383ece3c\") "
Sep 17 18:57:07.470415 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.470090    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00f2258-8e06-4e3a-8add-e39a383ece3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e00f2258-8e06-4e3a-8add-e39a383ece3c" (UID: "e00f2258-8e06-4e3a-8add-e39a383ece3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:07.470415 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.470204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e00f2258-8e06-4e3a-8add-e39a383ece3c" (UID: "e00f2258-8e06-4e3a-8add-e39a383ece3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:07.470415 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.470205    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e00f2258-8e06-4e3a-8add-e39a383ece3c" (UID: "e00f2258-8e06-4e3a-8add-e39a383ece3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:07.471784 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.471760    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-api-access-4fssr" (OuterVolumeSpecName: "kube-api-access-4fssr") pod "e00f2258-8e06-4e3a-8add-e39a383ece3c" (UID: "e00f2258-8e06-4e3a-8add-e39a383ece3c"). InnerVolumeSpecName "kube-api-access-4fssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:07.471871 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.471790    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e00f2258-8e06-4e3a-8add-e39a383ece3c" (UID: "e00f2258-8e06-4e3a-8add-e39a383ece3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:07.570820 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.570751    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:07.570820 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.570780    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f2258-8e06-4e3a-8add-e39a383ece3c-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:07.570820 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.570791    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fssr\" (UniqueName: \"kubernetes.io/projected/e00f2258-8e06-4e3a-8add-e39a383ece3c-kube-api-access-4fssr\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:07.570820 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.570799    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f2258-8e06-4e3a-8add-e39a383ece3c-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:07.570820 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.570807    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f2258-8e06-4e3a-8add-e39a383ece3c-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:07.856789 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.856709    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"]
Sep 17 18:57:07.857245 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.857027    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container" containerID="cri-o://10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112" gracePeriod=30
Sep 17 18:57:07.857245 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:07.857200    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kube-rbac-proxy" containerID="cri-o://452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d" gracePeriod=30
Sep 17 18:57:08.166787 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.166699    2565 generic.go:356] "Generic (PLEG): container finished" podID="1efd46be-4254-4685-8025-88b44bfd8380" containerID="452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d" exitCode=2
Sep 17 18:57:08.166787 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.166773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerDied","Data":"452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d"}
Sep 17 18:57:08.168541 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.168516    2565 generic.go:356] "Generic (PLEG): container finished" podID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerID="a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4" exitCode=0
Sep 17 18:57:08.168665 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.168545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerDied","Data":"a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4"}
Sep 17 18:57:08.168665 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.168576    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"
Sep 17 18:57:08.168665 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.168597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps" event={"ID":"e00f2258-8e06-4e3a-8add-e39a383ece3c","Type":"ContainerDied","Data":"a77861b13a4a483f373b06d69e08c18b3a5db357f58278a9f6f259cccd590944"}
Sep 17 18:57:08.168665 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.168624    2565 scope.go:117] "RemoveContainer" containerID="ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195"
Sep 17 18:57:08.177055 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.177035    2565 scope.go:117] "RemoveContainer" containerID="a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4"
Sep 17 18:57:08.183856 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.183839    2565 scope.go:117] "RemoveContainer" containerID="176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020"
Sep 17 18:57:08.190276 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.190255    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"]
Sep 17 18:57:08.190640 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.190619    2565 scope.go:117] "RemoveContainer" containerID="ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195"
Sep 17 18:57:08.190875 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:08.190857    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195\": container with ID starting with ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195 not found: ID does not exist" containerID="ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195"
Sep 17 18:57:08.190922 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.190884    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195"} err="failed to get container status \"ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195\": rpc error: code = NotFound desc = could not find container \"ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195\": container with ID starting with ee0bc781592981b3bc927c40d36befa56664978b689808247282130ddc4ac195 not found: ID does not exist"
Sep 17 18:57:08.190922 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.190900    2565 scope.go:117] "RemoveContainer" containerID="a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4"
Sep 17 18:57:08.191122 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:08.191107    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4\": container with ID starting with a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4 not found: ID does not exist" containerID="a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4"
Sep 17 18:57:08.191164 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.191127    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4"} err="failed to get container status \"a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4\": rpc error: code = NotFound desc = could not find container \"a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4\": container with ID starting with a7b2b2739390e745e9c19ec3c9303693ca9241873c1f473d094dca99ede030d4 not found: ID does not exist"
Sep 17 18:57:08.191164 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.191141    2565 scope.go:117] "RemoveContainer" containerID="176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020"
Sep 17 18:57:08.191382 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:08.191365    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020\": container with ID starting with 176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020 not found: ID does not exist" containerID="176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020"
Sep 17 18:57:08.191432 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.191390    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020"} err="failed to get container status \"176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020\": rpc error: code = NotFound desc = could not find container \"176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020\": container with ID starting with 176f816cc0b2e09125600e0c6147a170c110b0b6fd67f1af25d14141299b9020 not found: ID does not exist"
Sep 17 18:57:08.195482 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:08.195463    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-r74ps"]
Sep 17 18:57:09.061267 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:09.061238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" path="/var/lib/kubelet/pods/e00f2258-8e06-4e3a-8add-e39a383ece3c/volumes"
Sep 17 18:57:09.994639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:09.994608    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:57:10.816117 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:10.816078    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 18:57:12.087232 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.087208    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:57:12.100738 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.100717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-cabundle-cert\") pod \"1efd46be-4254-4685-8025-88b44bfd8380\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") "
Sep 17 18:57:12.100834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.100760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1efd46be-4254-4685-8025-88b44bfd8380-proxy-tls\") pod \"1efd46be-4254-4685-8025-88b44bfd8380\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") "
Sep 17 18:57:12.100834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.100783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1efd46be-4254-4685-8025-88b44bfd8380-kserve-provision-location\") pod \"1efd46be-4254-4685-8025-88b44bfd8380\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") "
Sep 17 18:57:12.100834 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.100800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgmfb\" (UniqueName: \"kubernetes.io/projected/1efd46be-4254-4685-8025-88b44bfd8380-kube-api-access-vgmfb\") pod \"1efd46be-4254-4685-8025-88b44bfd8380\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") "
Sep 17 18:57:12.100990 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.100854    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-kube-rbac-proxy-sar-config\") pod \"1efd46be-4254-4685-8025-88b44bfd8380\" (UID: \"1efd46be-4254-4685-8025-88b44bfd8380\") "
Sep 17 18:57:12.101049 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.101016    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1efd46be-4254-4685-8025-88b44bfd8380" (UID: "1efd46be-4254-4685-8025-88b44bfd8380"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:12.101096 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.101051    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efd46be-4254-4685-8025-88b44bfd8380-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1efd46be-4254-4685-8025-88b44bfd8380" (UID: "1efd46be-4254-4685-8025-88b44bfd8380"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:12.101233 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.101213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1efd46be-4254-4685-8025-88b44bfd8380" (UID: "1efd46be-4254-4685-8025-88b44bfd8380"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:12.102665 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.102641    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efd46be-4254-4685-8025-88b44bfd8380-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1efd46be-4254-4685-8025-88b44bfd8380" (UID: "1efd46be-4254-4685-8025-88b44bfd8380"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:12.102759 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.102660    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efd46be-4254-4685-8025-88b44bfd8380-kube-api-access-vgmfb" (OuterVolumeSpecName: "kube-api-access-vgmfb") pod "1efd46be-4254-4685-8025-88b44bfd8380" (UID: "1efd46be-4254-4685-8025-88b44bfd8380"). InnerVolumeSpecName "kube-api-access-vgmfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:12.184754 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.184727    2565 generic.go:356] "Generic (PLEG): container finished" podID="1efd46be-4254-4685-8025-88b44bfd8380" containerID="10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112" exitCode=0
Sep 17 18:57:12.184872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.184764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerDied","Data":"10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112"}
Sep 17 18:57:12.184872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.184783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"
Sep 17 18:57:12.184872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.184814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw" event={"ID":"1efd46be-4254-4685-8025-88b44bfd8380","Type":"ContainerDied","Data":"a76c62aa753af56120054bae0d9bc6a425046bb4b25da47a7bbcf92af8852937"}
Sep 17 18:57:12.184872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.184841    2565 scope.go:117] "RemoveContainer" containerID="452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d"
Sep 17 18:57:12.192932 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.192912    2565 scope.go:117] "RemoveContainer" containerID="10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112"
Sep 17 18:57:12.199721 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.199705    2565 scope.go:117] "RemoveContainer" containerID="d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5"
Sep 17 18:57:12.201772 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.201757    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.201849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.201775    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1efd46be-4254-4685-8025-88b44bfd8380-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.201849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.201784    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1efd46be-4254-4685-8025-88b44bfd8380-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.201849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.201791    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1efd46be-4254-4685-8025-88b44bfd8380-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.201849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.201800    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgmfb\" (UniqueName: \"kubernetes.io/projected/1efd46be-4254-4685-8025-88b44bfd8380-kube-api-access-vgmfb\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.206563 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.206466    2565 scope.go:117] "RemoveContainer" containerID="452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d"
Sep 17 18:57:12.207013 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:12.206989    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d\": container with ID starting with 452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d not found: ID does not exist" containerID="452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d"
Sep 17 18:57:12.207097 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.207020    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d"} err="failed to get container status \"452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d\": rpc error: code = NotFound desc = could not find container \"452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d\": container with ID starting with 452cbf969f51bf9cafa91edc62599aa1cd4e1e89901ce72ee29c394ee084303d not found: ID does not exist"
Sep 17 18:57:12.207097 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.207045    2565 scope.go:117] "RemoveContainer" containerID="10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112"
Sep 17 18:57:12.207317 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:12.207298    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112\": container with ID starting with 10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112 not found: ID does not exist" containerID="10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112"
Sep 17 18:57:12.207357 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.207321    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112"} err="failed to get container status \"10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112\": rpc error: code = NotFound desc = could not find container \"10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112\": container with ID starting with 10b565c4737e040ae9e795939c8fe4fb1330190ad0abf20afc5d2e252c793112 not found: ID does not exist"
Sep 17 18:57:12.207357 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.207335    2565 scope.go:117] "RemoveContainer" containerID="d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5"
Sep 17 18:57:12.207629 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:12.207611    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5\": container with ID starting with d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5 not found: ID does not exist" containerID="d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5"
Sep 17 18:57:12.207706 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.207629    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5"} err="failed to get container status \"d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5\": rpc error: code = NotFound desc = could not find container \"d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5\": container with ID starting with d4eb62106455bc2afed45f116057bf05ff9c73077ecdea0a566c406d3e5887f5 not found: ID does not exist"
Sep 17 18:57:12.208570 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.208551    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"]
Sep 17 18:57:12.214945 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:12.214922    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-fkdgw"]
Sep 17 18:57:13.062341 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:13.062307    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1efd46be-4254-4685-8025-88b44bfd8380" path="/var/lib/kubelet/pods/1efd46be-4254-4685-8025-88b44bfd8380/volumes"
Sep 17 18:57:18.148744 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.148714    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"]
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.148966    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kube-rbac-proxy"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.148978    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kube-rbac-proxy"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.148985    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.148991    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149002    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kube-rbac-proxy"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149007    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kube-rbac-proxy"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149012    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="storage-initializer"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149018    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="storage-initializer"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149034    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149042    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="storage-initializer"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149048    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="storage-initializer"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149053    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149058    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149102    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kube-rbac-proxy"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149109    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kube-rbac-proxy"
Sep 17 18:57:18.149106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149115    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efd46be-4254-4685-8025-88b44bfd8380" containerName="kserve-container"
Sep 17 18:57:18.149593 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00f2258-8e06-4e3a-8add-e39a383ece3c" containerName="kserve-container"
Sep 17 18:57:18.149593 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.149126    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e646460f-7720-4476-ae7c-836d695635bc" containerName="model-chainer-raw-bae88"
Sep 17 18:57:18.153569 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.153552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.157139 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.157123    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-bfe648-predictor-serving-cert\""
Sep 17 18:57:18.157367 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.157352    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-bfe648-kube-rbac-proxy-sar-config\""
Sep 17 18:57:18.166247 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.166227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"]
Sep 17 18:57:18.248700 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.248670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.248833 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.248712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d990187d-8d79-4559-bc2d-4414ed104292-kserve-provision-location\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.248833 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.248745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7h7\" (UniqueName: \"kubernetes.io/projected/d990187d-8d79-4559-bc2d-4414ed104292-kube-api-access-lb7h7\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.248833 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.248819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-cabundle-cert\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.248966 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.248858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.349570 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.349544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-cabundle-cert\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.349677 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.349573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.349677 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.349608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.349677 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.349631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d990187d-8d79-4559-bc2d-4414ed104292-kserve-provision-location\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.349677 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.349651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7h7\" (UniqueName: \"kubernetes.io/projected/d990187d-8d79-4559-bc2d-4414ed104292-kube-api-access-lb7h7\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.349850 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:18.349696    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-bfe648-predictor-serving-cert: secret "isvc-primary-bfe648-predictor-serving-cert" not found
Sep 17 18:57:18.349850 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:18.349750    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls podName:d990187d-8d79-4559-bc2d-4414ed104292 nodeName:}" failed. No retries permitted until 2026-09-17 18:57:18.849732053 +0000 UTC m=+1128.136044547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls") pod "isvc-primary-bfe648-predictor-686895584-c5vq5" (UID: "d990187d-8d79-4559-bc2d-4414ed104292") : secret "isvc-primary-bfe648-predictor-serving-cert" not found
Sep 17 18:57:18.350053 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.350033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d990187d-8d79-4559-bc2d-4414ed104292-kserve-provision-location\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.350262 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.350247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.350297 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.350252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-cabundle-cert\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.361793 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.361770    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7h7\" (UniqueName: \"kubernetes.io/projected/d990187d-8d79-4559-bc2d-4414ed104292-kube-api-access-lb7h7\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.853824 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.853786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:18.855986 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:18.855964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls\") pod \"isvc-primary-bfe648-predictor-686895584-c5vq5\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") " pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:19.062592 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:19.062567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:19.335164 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:19.335094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"]
Sep 17 18:57:19.337819 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:57:19.337780    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd990187d_8d79_4559_bc2d_4414ed104292.slice/crio-c6cfbf6087301e03e582d90f73010a0b6069947f2c8c6ad26fb81df2eb4385ee WatchSource:0}: Error finding container c6cfbf6087301e03e582d90f73010a0b6069947f2c8c6ad26fb81df2eb4385ee: Status 404 returned error can't find the container with id c6cfbf6087301e03e582d90f73010a0b6069947f2c8c6ad26fb81df2eb4385ee
Sep 17 18:57:19.995067 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:19.995028    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:57:20.217991 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:20.217958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerStarted","Data":"a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba"}
Sep 17 18:57:20.218140 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:20.218017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerStarted","Data":"c6cfbf6087301e03e582d90f73010a0b6069947f2c8c6ad26fb81df2eb4385ee"}
Sep 17 18:57:24.233662 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:24.233631    2565 generic.go:356] "Generic (PLEG): container finished" podID="d990187d-8d79-4559-bc2d-4414ed104292" containerID="a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba" exitCode=0
Sep 17 18:57:24.234009 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:24.233676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerDied","Data":"a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba"}
Sep 17 18:57:25.239178 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:25.239142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerStarted","Data":"3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323"}
Sep 17 18:57:25.239547 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:25.239202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerStarted","Data":"738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab"}
Sep 17 18:57:25.239547 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:25.239380    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:25.258810 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:25.258767    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podStartSLOduration=3.155997302 podStartE2EDuration="7.2587543s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.102756998s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:25.257751675 +0000 UTC m=+1134.544064229" watchObservedRunningTime="2026-09-17 18:57:25.2587543 +0000 UTC m=+1134.545066816"
Sep 17 18:57:26.243327 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:26.243296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:26.244642 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:26.244617    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:57:27.247806 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:27.247765    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:57:29.996380 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:29.996343    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:57:32.252886 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:32.252858    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:57:32.253317 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:32.253257    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:57:38.173875 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.173840    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"]
Sep 17 18:57:38.178524 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.178503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.181284 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.181266    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-8b7b7-kube-rbac-proxy-sar-config\""
Sep 17 18:57:38.181366 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.181296    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-8b7b7-serving-cert\""
Sep 17 18:57:38.190480 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.190461    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"]
Sep 17 18:57:38.286924 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.286895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.287034 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.286945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6bd491-8e7c-4410-bdc6-82746a6331fd-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.387452 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.387415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6bd491-8e7c-4410-bdc6-82746a6331fd-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.387545 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.387491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.387617 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:38.387601    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-serving-cert: secret "model-chainer-raw-hpa-8b7b7-serving-cert" not found
Sep 17 18:57:38.387687 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:57:38.387677    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls podName:6a6bd491-8e7c-4410-bdc6-82746a6331fd nodeName:}" failed. No retries permitted until 2026-09-17 18:57:38.887656033 +0000 UTC m=+1148.173968529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls") pod "model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" (UID: "6a6bd491-8e7c-4410-bdc6-82746a6331fd") : secret "model-chainer-raw-hpa-8b7b7-serving-cert" not found
Sep 17 18:57:38.388101 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.388084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6bd491-8e7c-4410-bdc6-82746a6331fd-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.890787 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.890755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:38.893101 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:38.893071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls\") pod \"model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:39.088294 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:39.088266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:39.361757 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:39.361735    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"]
Sep 17 18:57:39.364104 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:57:39.364073    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6bd491_8e7c_4410_bdc6_82746a6331fd.slice/crio-55bfffa252701776b9177d62b1998bbb3761bc9dfd9a89ac60de1dd3e1742932 WatchSource:0}: Error finding container 55bfffa252701776b9177d62b1998bbb3761bc9dfd9a89ac60de1dd3e1742932: Status 404 returned error can't find the container with id 55bfffa252701776b9177d62b1998bbb3761bc9dfd9a89ac60de1dd3e1742932
Sep 17 18:57:40.294661 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:40.294627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" event={"ID":"6a6bd491-8e7c-4410-bdc6-82746a6331fd","Type":"ContainerStarted","Data":"5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2"}
Sep 17 18:57:40.294849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:40.294686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" event={"ID":"6a6bd491-8e7c-4410-bdc6-82746a6331fd","Type":"ContainerStarted","Data":"55bfffa252701776b9177d62b1998bbb3761bc9dfd9a89ac60de1dd3e1742932"}
Sep 17 18:57:40.294849 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:40.294745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:40.325297 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:40.325253    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podStartSLOduration=2.325240514 podStartE2EDuration="2.325240514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:40.324188679 +0000 UTC m=+1149.610501195" watchObservedRunningTime="2026-09-17 18:57:40.325240514 +0000 UTC m=+1149.611553030"
Sep 17 18:57:42.253970 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:42.253932    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:57:46.303755 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:46.303678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:57:48.212739 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:48.212705    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"]
Sep 17 18:57:48.213112 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:48.212890    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" containerID="cri-o://5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2" gracePeriod=30
Sep 17 18:57:48.340085 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:48.340043    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"]
Sep 17 18:57:48.340380 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:48.340350    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" containerID="cri-o://b51fc2bd7c38d1815453e9a210a982818260f7c28f5778a2eecb0b5a590391be" gracePeriod=30
Sep 17 18:57:48.340498 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:48.340418    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kube-rbac-proxy" containerID="cri-o://415f345c87f01ff204f96bd0bff6df4e3d1c2c7cdef7f05acf8a65ddbb9a774e" gracePeriod=30
Sep 17 18:57:49.331236 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:49.331201    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerID="415f345c87f01ff204f96bd0bff6df4e3d1c2c7cdef7f05acf8a65ddbb9a774e" exitCode=2
Sep 17 18:57:49.331640 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:49.331273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerDied","Data":"415f345c87f01ff204f96bd0bff6df4e3d1c2c7cdef7f05acf8a65ddbb9a774e"}
Sep 17 18:57:49.989878 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:49.989840    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:57:49.995234 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:49.995202    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:57:51.302548 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:51.302502    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:57:52.253825 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.253790    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:57:52.347071 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.347037    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerID="b51fc2bd7c38d1815453e9a210a982818260f7c28f5778a2eecb0b5a590391be" exitCode=0
Sep 17 18:57:52.347404 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.347107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerDied","Data":"b51fc2bd7c38d1815453e9a210a982818260f7c28f5778a2eecb0b5a590391be"}
Sep 17 18:57:52.368636 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.368614    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:57:52.491989 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.491969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxcjx\" (UniqueName: \"kubernetes.io/projected/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-api-access-dxcjx\") pod \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") "
Sep 17 18:57:52.492129 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2d5a63-885b-406b-bddc-480cdb1a73c0-proxy-tls\") pod \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") "
Sep 17 18:57:52.492129 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-cabundle-cert\") pod \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") "
Sep 17 18:57:52.492241 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-rbac-proxy-sar-config\") pod \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") "
Sep 17 18:57:52.492241 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kserve-provision-location\") pod \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\" (UID: \"6a2d5a63-885b-406b-bddc-480cdb1a73c0\") "
Sep 17 18:57:52.492339 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492316    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a2d5a63-885b-406b-bddc-480cdb1a73c0" (UID: "6a2d5a63-885b-406b-bddc-480cdb1a73c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:52.492566 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a2d5a63-885b-406b-bddc-480cdb1a73c0" (UID: "6a2d5a63-885b-406b-bddc-480cdb1a73c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:52.492653 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a2d5a63-885b-406b-bddc-480cdb1a73c0" (UID: "6a2d5a63-885b-406b-bddc-480cdb1a73c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:52.492653 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.492572    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.494025 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.494001    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-api-access-dxcjx" (OuterVolumeSpecName: "kube-api-access-dxcjx") pod "6a2d5a63-885b-406b-bddc-480cdb1a73c0" (UID: "6a2d5a63-885b-406b-bddc-480cdb1a73c0"). InnerVolumeSpecName "kube-api-access-dxcjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:52.494113 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.494036    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2d5a63-885b-406b-bddc-480cdb1a73c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a2d5a63-885b-406b-bddc-480cdb1a73c0" (UID: "6a2d5a63-885b-406b-bddc-480cdb1a73c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:52.593520 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.593490    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.593520 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.593514    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.593520 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.593523    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcjx\" (UniqueName: \"kubernetes.io/projected/6a2d5a63-885b-406b-bddc-480cdb1a73c0-kube-api-access-dxcjx\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.593720 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:52.593533    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2d5a63-885b-406b-bddc-480cdb1a73c0-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:57:53.352982 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.352899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr" event={"ID":"6a2d5a63-885b-406b-bddc-480cdb1a73c0","Type":"ContainerDied","Data":"e254ab2590f17ed8b8b0b07549caee1a867df81d9233b31dce5e94f8a252b177"}
Sep 17 18:57:53.352982 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.352969    2565 scope.go:117] "RemoveContainer" containerID="415f345c87f01ff204f96bd0bff6df4e3d1c2c7cdef7f05acf8a65ddbb9a774e"
Sep 17 18:57:53.353431 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.352910    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"
Sep 17 18:57:53.360666 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.360647    2565 scope.go:117] "RemoveContainer" containerID="b51fc2bd7c38d1815453e9a210a982818260f7c28f5778a2eecb0b5a590391be"
Sep 17 18:57:53.367704 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.367686    2565 scope.go:117] "RemoveContainer" containerID="0cf13c014e1a31601acc9847848f556e43f95e09d4b157e860ff88bb19c20f27"
Sep 17 18:57:53.382949 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.382917    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"]
Sep 17 18:57:53.384982 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:53.384963    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-8b7b7-predictor-54bd6475d9-ddzjr"]
Sep 17 18:57:55.061530 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:55.061500    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" path="/var/lib/kubelet/pods/6a2d5a63-885b-406b-bddc-480cdb1a73c0/volumes"
Sep 17 18:57:56.302867 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:57:56.302829    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:01.302705 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:01.302667    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:01.303113 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:01.302778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:58:02.253348 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:02.253312    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:58:06.303054 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:06.303014    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484001    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"]
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484699    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484719    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484740    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="storage-initializer"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484749    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="storage-initializer"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484770    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kube-rbac-proxy"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484779    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kube-rbac-proxy"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484939    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kube-rbac-proxy"
Sep 17 18:58:08.486186 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.484951    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2d5a63-885b-406b-bddc-480cdb1a73c0" containerName="kserve-container"
Sep 17 18:58:08.491931 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.491904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.494344 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.494322    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-47a1c-predictor-serving-cert\""
Sep 17 18:58:08.494344 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.494335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-47a1c-kube-rbac-proxy-sar-config\""
Sep 17 18:58:08.502151 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.502129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"]
Sep 17 18:58:08.619608 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.619572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.619781 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.619617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58b20c8-553f-445e-9dc0-917711c06777-kserve-provision-location\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.619781 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.619674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-cabundle-cert\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.619781 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.619714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.619781 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.619740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286ms\" (UniqueName: \"kubernetes.io/projected/a58b20c8-553f-445e-9dc0-917711c06777-kube-api-access-286ms\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.720956 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.720923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.720965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58b20c8-553f-445e-9dc0-917711c06777-kserve-provision-location\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.720982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-cabundle-cert\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.721006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721269 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:58:08.721108    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-serving-cert: secret "isvc-logger-raw-47a1c-predictor-serving-cert" not found
Sep 17 18:58:08.721269 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.721124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286ms\" (UniqueName: \"kubernetes.io/projected/a58b20c8-553f-445e-9dc0-917711c06777-kube-api-access-286ms\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721269 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:58:08.721179    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls podName:a58b20c8-553f-445e-9dc0-917711c06777 nodeName:}" failed. No retries permitted until 2026-09-17 18:58:09.221159518 +0000 UTC m=+1178.507472028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls") pod "isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" (UID: "a58b20c8-553f-445e-9dc0-917711c06777") : secret "isvc-logger-raw-47a1c-predictor-serving-cert" not found
Sep 17 18:58:08.721421 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.721345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58b20c8-553f-445e-9dc0-917711c06777-kserve-provision-location\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721705 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.721687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-cabundle-cert\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.721742 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.721700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:08.732048 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:08.732027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-286ms\" (UniqueName: \"kubernetes.io/projected/a58b20c8-553f-445e-9dc0-917711c06777-kube-api-access-286ms\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:09.226055 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:09.226021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:09.228541 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:09.228509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls\") pod \"isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:09.401308 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:09.401272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:09.682349 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:09.682321    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"]
Sep 17 18:58:09.684651 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:58:09.684623    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58b20c8_553f_445e_9dc0_917711c06777.slice/crio-77409085c8e7ad776db4a191a2103b85db6833727a89344e5661fcf45d1108ea WatchSource:0}: Error finding container 77409085c8e7ad776db4a191a2103b85db6833727a89344e5661fcf45d1108ea: Status 404 returned error can't find the container with id 77409085c8e7ad776db4a191a2103b85db6833727a89344e5661fcf45d1108ea
Sep 17 18:58:10.423638 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:10.423603    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerStarted","Data":"690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa"}
Sep 17 18:58:10.423803 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:10.423672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerStarted","Data":"77409085c8e7ad776db4a191a2103b85db6833727a89344e5661fcf45d1108ea"}
Sep 17 18:58:11.302540 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:11.302507    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:12.254217 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:12.254181    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:58:14.441362 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:14.441328    2565 generic.go:356] "Generic (PLEG): container finished" podID="a58b20c8-553f-445e-9dc0-917711c06777" containerID="690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa" exitCode=0
Sep 17 18:58:14.441728 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:14.441372    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerDied","Data":"690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa"}
Sep 17 18:58:15.450260 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.450224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerStarted","Data":"0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc"}
Sep 17 18:58:15.450729 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.450288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerStarted","Data":"86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b"}
Sep 17 18:58:15.450729 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.450312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerStarted","Data":"150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e"}
Sep 17 18:58:15.450729 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.450601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:15.450869 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.450787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:15.450869 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.450834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:15.452193 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.452164    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:58:15.452843 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.452824    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:15.477412 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:15.477367    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podStartSLOduration=3.632271737 podStartE2EDuration="7.477354082s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845082345s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:15.475306807 +0000 UTC m=+1184.761619324" watchObservedRunningTime="2026-09-17 18:58:15.477354082 +0000 UTC m=+1184.763666599"
Sep 17 18:58:16.302550 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:16.302515    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:16.455143 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:16.455105    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:58:16.455573 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:16.455457    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:18.354685 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.354659    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:58:18.464579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.463897    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerID="5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2" exitCode=0
Sep 17 18:58:18.464579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.463998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" event={"ID":"6a6bd491-8e7c-4410-bdc6-82746a6331fd","Type":"ContainerDied","Data":"5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2"}
Sep 17 18:58:18.464579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.464079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs" event={"ID":"6a6bd491-8e7c-4410-bdc6-82746a6331fd","Type":"ContainerDied","Data":"55bfffa252701776b9177d62b1998bbb3761bc9dfd9a89ac60de1dd3e1742932"}
Sep 17 18:58:18.464579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.464106    2565 scope.go:117] "RemoveContainer" containerID="5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2"
Sep 17 18:58:18.464579 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.464253    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"
Sep 17 18:58:18.473170 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.473150    2565 scope.go:117] "RemoveContainer" containerID="5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2"
Sep 17 18:58:18.473454 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:58:18.473419    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2\": container with ID starting with 5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2 not found: ID does not exist" containerID="5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2"
Sep 17 18:58:18.473507 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.473464    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2"} err="failed to get container status \"5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2\": rpc error: code = NotFound desc = could not find container \"5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2\": container with ID starting with 5f79a9e486496cfc6bec69d0888eb1c09e8c109f0efd64ce4f9d8bd0c872dec2 not found: ID does not exist"
Sep 17 18:58:18.498209 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.498187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls\") pod \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") "
Sep 17 18:58:18.498282 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.498247    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6bd491-8e7c-4410-bdc6-82746a6331fd-openshift-service-ca-bundle\") pod \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\" (UID: \"6a6bd491-8e7c-4410-bdc6-82746a6331fd\") "
Sep 17 18:58:18.498531 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.498511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6bd491-8e7c-4410-bdc6-82746a6331fd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6a6bd491-8e7c-4410-bdc6-82746a6331fd" (UID: "6a6bd491-8e7c-4410-bdc6-82746a6331fd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:58:18.500161 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.500143    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a6bd491-8e7c-4410-bdc6-82746a6331fd" (UID: "6a6bd491-8e7c-4410-bdc6-82746a6331fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:58:18.599217 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.599188    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6bd491-8e7c-4410-bdc6-82746a6331fd-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:58:18.599217 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.599215    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6bd491-8e7c-4410-bdc6-82746a6331fd-openshift-service-ca-bundle\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:58:18.785091 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.785064    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"]
Sep 17 18:58:18.788641 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:18.788621    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-8b7b7-778bc8d777-68xhs"]
Sep 17 18:58:19.061340 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:19.061266    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" path="/var/lib/kubelet/pods/6a6bd491-8e7c-4410-bdc6-82746a6331fd/volumes"
Sep 17 18:58:21.459529 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:21.459502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:58:21.460115 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:21.460080    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:58:21.460272 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:21.460249    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:22.253452 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:22.253408    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:58:31.460666 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:31.460615    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:58:31.461131 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:31.461060    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:32.254216 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:32.254176    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:58:41.460692 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:41.460657    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:58:41.461235 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:41.461140    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:58:42.255139 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:42.255111    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:58:51.460775 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:51.460725    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:58:51.461224 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:58:51.461203    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:59:01.460018 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:01.459980    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:59:01.460454 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:01.460403    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:59:11.460599 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:11.460555    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:59:11.461038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:11.460997    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:59:17.455953 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:17.455881    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"]
Sep 17 18:59:17.456379 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:17.456148    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container" containerID="cri-o://738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab" gracePeriod=30
Sep 17 18:59:17.456379 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:17.456186    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kube-rbac-proxy" containerID="cri-o://3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323" gracePeriod=30
Sep 17 18:59:17.686718 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:17.686687    2565 generic.go:356] "Generic (PLEG): container finished" podID="d990187d-8d79-4559-bc2d-4414ed104292" containerID="3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323" exitCode=2
Sep 17 18:59:17.686844 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:17.686749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerDied","Data":"3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323"}
Sep 17 18:59:21.460613 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.460577    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:59:21.461084 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.461061    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:59:21.499650 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.499630    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:59:21.548656 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.548634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb7h7\" (UniqueName: \"kubernetes.io/projected/d990187d-8d79-4559-bc2d-4414ed104292-kube-api-access-lb7h7\") pod \"d990187d-8d79-4559-bc2d-4414ed104292\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") "
Sep 17 18:59:21.548801 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.548688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls\") pod \"d990187d-8d79-4559-bc2d-4414ed104292\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") "
Sep 17 18:59:21.548872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.548815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d990187d-8d79-4559-bc2d-4414ed104292-kserve-provision-location\") pod \"d990187d-8d79-4559-bc2d-4414ed104292\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") "
Sep 17 18:59:21.548872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.548857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-kube-rbac-proxy-sar-config\") pod \"d990187d-8d79-4559-bc2d-4414ed104292\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") "
Sep 17 18:59:21.548980 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.548912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-cabundle-cert\") pod \"d990187d-8d79-4559-bc2d-4414ed104292\" (UID: \"d990187d-8d79-4559-bc2d-4414ed104292\") "
Sep 17 18:59:21.549169 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.549148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d990187d-8d79-4559-bc2d-4414ed104292-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d990187d-8d79-4559-bc2d-4414ed104292" (UID: "d990187d-8d79-4559-bc2d-4414ed104292"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:21.549306 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.549279    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d990187d-8d79-4559-bc2d-4414ed104292" (UID: "d990187d-8d79-4559-bc2d-4414ed104292"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:21.549410 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.549382    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d990187d-8d79-4559-bc2d-4414ed104292" (UID: "d990187d-8d79-4559-bc2d-4414ed104292"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:21.550825 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.550798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d990187d-8d79-4559-bc2d-4414ed104292-kube-api-access-lb7h7" (OuterVolumeSpecName: "kube-api-access-lb7h7") pod "d990187d-8d79-4559-bc2d-4414ed104292" (UID: "d990187d-8d79-4559-bc2d-4414ed104292"). InnerVolumeSpecName "kube-api-access-lb7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:21.550971 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.550949    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d990187d-8d79-4559-bc2d-4414ed104292" (UID: "d990187d-8d79-4559-bc2d-4414ed104292"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:21.649967 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.649909    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7h7\" (UniqueName: \"kubernetes.io/projected/d990187d-8d79-4559-bc2d-4414ed104292-kube-api-access-lb7h7\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.649967 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.649932    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d990187d-8d79-4559-bc2d-4414ed104292-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.649967 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.649946    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d990187d-8d79-4559-bc2d-4414ed104292-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.649967 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.649958    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.650146 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.649971    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d990187d-8d79-4559-bc2d-4414ed104292-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.703609 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.703583    2565 generic.go:356] "Generic (PLEG): container finished" podID="d990187d-8d79-4559-bc2d-4414ed104292" containerID="738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab" exitCode=0
Sep 17 18:59:21.703689 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.703633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerDied","Data":"738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab"}
Sep 17 18:59:21.703689 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.703644    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"
Sep 17 18:59:21.703772 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.703728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5" event={"ID":"d990187d-8d79-4559-bc2d-4414ed104292","Type":"ContainerDied","Data":"c6cfbf6087301e03e582d90f73010a0b6069947f2c8c6ad26fb81df2eb4385ee"}
Sep 17 18:59:21.703772 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.703768    2565 scope.go:117] "RemoveContainer" containerID="3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323"
Sep 17 18:59:21.712614 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.712595    2565 scope.go:117] "RemoveContainer" containerID="738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab"
Sep 17 18:59:21.719710 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.719695    2565 scope.go:117] "RemoveContainer" containerID="a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba"
Sep 17 18:59:21.726756 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.726737    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"]
Sep 17 18:59:21.727378 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.727368    2565 scope.go:117] "RemoveContainer" containerID="3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323"
Sep 17 18:59:21.727680 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:59:21.727662    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323\": container with ID starting with 3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323 not found: ID does not exist" containerID="3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323"
Sep 17 18:59:21.727725 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.727689    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323"} err="failed to get container status \"3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323\": rpc error: code = NotFound desc = could not find container \"3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323\": container with ID starting with 3826f250d781453078b7d6316d107755d7b93fc2fb0594c6103122a9ef8d2323 not found: ID does not exist"
Sep 17 18:59:21.727725 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.727705    2565 scope.go:117] "RemoveContainer" containerID="738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab"
Sep 17 18:59:21.727962 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:59:21.727944    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab\": container with ID starting with 738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab not found: ID does not exist" containerID="738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab"
Sep 17 18:59:21.728030 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.727971    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab"} err="failed to get container status \"738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab\": rpc error: code = NotFound desc = could not find container \"738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab\": container with ID starting with 738904b22217e5ccc033948153333f8dbb2a1b6844a16ea69d8b896d754312ab not found: ID does not exist"
Sep 17 18:59:21.728030 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.727992    2565 scope.go:117] "RemoveContainer" containerID="a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba"
Sep 17 18:59:21.728258 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:59:21.728242    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba\": container with ID starting with a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba not found: ID does not exist" containerID="a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba"
Sep 17 18:59:21.728297 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.728262    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba"} err="failed to get container status \"a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba\": rpc error: code = NotFound desc = could not find container \"a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba\": container with ID starting with a5dbe94a904e93df43c56054808e393d639c4c861e03e1b82ae2adc2b98b8aba not found: ID does not exist"
Sep 17 18:59:21.733038 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:21.733018    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bfe648-predictor-686895584-c5vq5"]
Sep 17 18:59:22.694166 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"]
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694411    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694422    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694428    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kube-rbac-proxy"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694447    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kube-rbac-proxy"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694460    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="storage-initializer"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694466    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="storage-initializer"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694478    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694484    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694526    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kserve-container"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694535    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6bd491-8e7c-4410-bdc6-82746a6331fd" containerName="model-chainer-raw-hpa-8b7b7"
Sep 17 18:59:22.694557 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.694543    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d990187d-8d79-4559-bc2d-4414ed104292" containerName="kube-rbac-proxy"
Sep 17 18:59:22.699050 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.699033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.717741 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.717714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"]
Sep 17 18:59:22.722686 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.722663    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-ca8764\""
Sep 17 18:59:22.723021 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.723004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-ca8764-kube-rbac-proxy-sar-config\""
Sep 17 18:59:22.723544 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.723530    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-ca8764-predictor-serving-cert\""
Sep 17 18:59:22.725296 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.725274    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-ca8764-dockercfg-q5rvr\""
Sep 17 18:59:22.757473 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.757429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.757567 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.757493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-cabundle-cert\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.757567 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.757550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-api-access-q92ps\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.757639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.757583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-proxy-tls\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.757639 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.757613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kserve-provision-location\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.858840 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.858815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-cabundle-cert\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.858933 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.858848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-api-access-q92ps\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.858933 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.858865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-proxy-tls\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.858933 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.858884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kserve-provision-location\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.859044 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.859011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.859318 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.859296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kserve-provision-location\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.859510 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.859492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-cabundle-cert\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.859660 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.859639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.861247 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.861229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-proxy-tls\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:22.868696 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:22.868679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-api-access-q92ps\") pod \"isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") " pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:23.009381 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:23.009331    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:23.064094 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:23.064067    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d990187d-8d79-4559-bc2d-4414ed104292" path="/var/lib/kubelet/pods/d990187d-8d79-4559-bc2d-4414ed104292/volumes"
Sep 17 18:59:23.305840 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:23.305763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"]
Sep 17 18:59:23.310849 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:59:23.310821    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d6a2b8_fc92_41a6_9fcc_fa7e7745d86d.slice/crio-b574fd5e71d979677307e8e0be861075d21d76bc73268b18d8877eea86719221 WatchSource:0}: Error finding container b574fd5e71d979677307e8e0be861075d21d76bc73268b18d8877eea86719221: Status 404 returned error can't find the container with id b574fd5e71d979677307e8e0be861075d21d76bc73268b18d8877eea86719221
Sep 17 18:59:23.714872 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:23.714833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" event={"ID":"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d","Type":"ContainerStarted","Data":"32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c"}
Sep 17 18:59:23.715239 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:23.714920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" event={"ID":"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d","Type":"ContainerStarted","Data":"b574fd5e71d979677307e8e0be861075d21d76bc73268b18d8877eea86719221"}
Sep 17 18:59:28.734487 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:28.734461    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx_b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/storage-initializer/0.log"
Sep 17 18:59:28.734794 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:28.734499    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerID="32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c" exitCode=1
Sep 17 18:59:28.734794 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:28.734560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" event={"ID":"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d","Type":"ContainerDied","Data":"32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c"}
Sep 17 18:59:29.739895 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:29.739871    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx_b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/storage-initializer/0.log"
Sep 17 18:59:29.740260 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:29.739987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" event={"ID":"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d","Type":"ContainerStarted","Data":"34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3"}
Sep 17 18:59:31.461493 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:31.461432    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:59:31.461868 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:31.461566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 18:59:32.747028 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:32.746994    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"]
Sep 17 18:59:32.747481 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:32.747314    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer" containerID="cri-o://34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3" gracePeriod=30
Sep 17 18:59:33.285563 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.285541    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx_b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/storage-initializer/1.log"
Sep 17 18:59:33.285974 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.285957    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx_b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/storage-initializer/0.log"
Sep 17 18:59:33.288348 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.288335    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:33.338930 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.338868    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-rbac-proxy-sar-config\") pod \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") "
Sep 17 18:59:33.338930 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.338907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-cabundle-cert\") pod \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") "
Sep 17 18:59:33.339099 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.338958    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-proxy-tls\") pod \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") "
Sep 17 18:59:33.339099 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.339003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-api-access-q92ps\") pod \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") "
Sep 17 18:59:33.339099 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.339039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kserve-provision-location\") pod \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\" (UID: \"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d\") "
Sep 17 18:59:33.339260 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.339189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" (UID: "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:33.339317 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.339292    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" (UID: "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:33.339413 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.339387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" (UID: "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:33.341493 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.341466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-api-access-q92ps" (OuterVolumeSpecName: "kube-api-access-q92ps") pod "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" (UID: "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d"). InnerVolumeSpecName "kube-api-access-q92ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:33.341493 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.341478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" (UID: "b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:33.439906 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.439870    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:33.439906 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.439904    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:33.439906 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.439915    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:33.440062 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.439922    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kube-api-access-q92ps\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:33.440062 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.439931    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 18:59:33.755165 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755142    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx_b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/storage-initializer/1.log"
Sep 17 18:59:33.755602 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755588    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx_b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/storage-initializer/0.log"
Sep 17 18:59:33.755646 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755621    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerID="34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3" exitCode=1
Sep 17 18:59:33.755699 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" event={"ID":"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d","Type":"ContainerDied","Data":"34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3"}
Sep 17 18:59:33.755699 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755682    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"
Sep 17 18:59:33.755762 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx" event={"ID":"b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d","Type":"ContainerDied","Data":"b574fd5e71d979677307e8e0be861075d21d76bc73268b18d8877eea86719221"}
Sep 17 18:59:33.755762 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.755746    2565 scope.go:117] "RemoveContainer" containerID="34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3"
Sep 17 18:59:33.764656 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.764639    2565 scope.go:117] "RemoveContainer" containerID="32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c"
Sep 17 18:59:33.771610 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.771593    2565 scope.go:117] "RemoveContainer" containerID="34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3"
Sep 17 18:59:33.771852 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:59:33.771833    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3\": container with ID starting with 34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3 not found: ID does not exist" containerID="34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3"
Sep 17 18:59:33.771899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.771860    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3"} err="failed to get container status \"34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3\": rpc error: code = NotFound desc = could not find container \"34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3\": container with ID starting with 34c97ab5e1299922a0f49dfa64b6529af2bc55b9b5202d8501446f00af942fe3 not found: ID does not exist"
Sep 17 18:59:33.771899 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.771876    2565 scope.go:117] "RemoveContainer" containerID="32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c"
Sep 17 18:59:33.772134 ip-10-0-130-95 kubenswrapper[2565]: E0917 18:59:33.772116    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c\": container with ID starting with 32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c not found: ID does not exist" containerID="32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c"
Sep 17 18:59:33.772178 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.772141    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c"} err="failed to get container status \"32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c\": rpc error: code = NotFound desc = could not find container \"32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c\": container with ID starting with 32a5fe691b50a5c56980d00eda58cd520f5ce1e2a72440140c11382779c9a64c not found: ID does not exist"
Sep 17 18:59:33.815076 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.815051    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"]
Sep 17 18:59:33.816295 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:33.816277    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ca8764-predictor-5b9cfb67d7-lkttx"]
Sep 17 18:59:35.062111 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:35.062078    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" path="/var/lib/kubelet/pods/b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d/volumes"
Sep 17 18:59:38.112181 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112151    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"]
Sep 17 18:59:38.112552 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112455    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer"
Sep 17 18:59:38.112552 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112468    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer"
Sep 17 18:59:38.112552 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112484    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer"
Sep 17 18:59:38.112552 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112489    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer"
Sep 17 18:59:38.112552 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112551    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer"
Sep 17 18:59:38.112702 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.112654    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d6a2b8-fc92-41a6-9fcc-fa7e7745d86d" containerName="storage-initializer"
Sep 17 18:59:38.117001 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.116980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.119928 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.119908    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1cac1-predictor-serving-cert\""
Sep 17 18:59:38.120209 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.120192    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1cac1-kube-rbac-proxy-sar-config\""
Sep 17 18:59:38.144935 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.144915    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"]
Sep 17 18:59:38.172006 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.171982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.172099 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.172014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnf69\" (UniqueName: \"kubernetes.io/projected/ad6d82cc-a179-4695-a910-705de69a1559-kube-api-access-cnf69\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.172099 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.172051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad6d82cc-a179-4695-a910-705de69a1559-proxy-tls\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.172176 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.172096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad6d82cc-a179-4695-a910-705de69a1559-kserve-provision-location\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.172176 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.172119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-cabundle-cert\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273068 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273171 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnf69\" (UniqueName: \"kubernetes.io/projected/ad6d82cc-a179-4695-a910-705de69a1559-kube-api-access-cnf69\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273171 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad6d82cc-a179-4695-a910-705de69a1559-proxy-tls\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273260 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad6d82cc-a179-4695-a910-705de69a1559-kserve-provision-location\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273308 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-cabundle-cert\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273658 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad6d82cc-a179-4695-a910-705de69a1559-kserve-provision-location\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273840 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-cabundle-cert\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.273840 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.273834    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.275487 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.275460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad6d82cc-a179-4695-a910-705de69a1559-proxy-tls\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.285607 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.285586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnf69\" (UniqueName: \"kubernetes.io/projected/ad6d82cc-a179-4695-a910-705de69a1559-kube-api-access-cnf69\") pod \"raw-sklearn-1cac1-predictor-598495cdf-kxwc8\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") " pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.426585 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.426527    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:38.699271 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.699198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"]
Sep 17 18:59:38.701450 ip-10-0-130-95 kubenswrapper[2565]: W0917 18:59:38.701412    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6d82cc_a179_4695_a910_705de69a1559.slice/crio-0647758bfffbf367b29ddddbc7e8353916798b97950e226f032d69d39d4e5325 WatchSource:0}: Error finding container 0647758bfffbf367b29ddddbc7e8353916798b97950e226f032d69d39d4e5325: Status 404 returned error can't find the container with id 0647758bfffbf367b29ddddbc7e8353916798b97950e226f032d69d39d4e5325
Sep 17 18:59:38.775671 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.775644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerStarted","Data":"48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9"}
Sep 17 18:59:38.775783 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:38.775704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerStarted","Data":"0647758bfffbf367b29ddddbc7e8353916798b97950e226f032d69d39d4e5325"}
Sep 17 18:59:42.792106 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:42.792072    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad6d82cc-a179-4695-a910-705de69a1559" containerID="48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9" exitCode=0
Sep 17 18:59:42.792469 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:42.792147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerDied","Data":"48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9"}
Sep 17 18:59:43.743275 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.743241    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"]
Sep 17 18:59:43.743542 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.743508    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" containerID="cri-o://150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e" gracePeriod=30
Sep 17 18:59:43.743542 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.743521    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" containerID="cri-o://0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc" gracePeriod=30
Sep 17 18:59:43.743661 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.743530    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" containerID="cri-o://86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b" gracePeriod=30
Sep 17 18:59:43.803976 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.803941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerStarted","Data":"f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863"}
Sep 17 18:59:43.804391 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.804043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerStarted","Data":"4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79"}
Sep 17 18:59:43.804391 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.804266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:43.859358 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:43.859317    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podStartSLOduration=1.9706562779999999 podStartE2EDuration="5.859302984s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.888646706s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:43.858780752 +0000 UTC m=+1273.145093271" watchObservedRunningTime="2026-09-17 18:59:43.859302984 +0000 UTC m=+1273.145615501"
Sep 17 18:59:44.810631 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:44.810602    2565 generic.go:356] "Generic (PLEG): container finished" podID="a58b20c8-553f-445e-9dc0-917711c06777" containerID="86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b" exitCode=2
Sep 17 18:59:44.811039 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:44.810676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerDied","Data":"86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b"}
Sep 17 18:59:44.811039 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:44.810935    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:44.812389 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:44.812367    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:59:45.814350 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:45.814302    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:59:46.456044 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:46.456005    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:59:48.828414 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:48.828382    2565 generic.go:356] "Generic (PLEG): container finished" podID="a58b20c8-553f-445e-9dc0-917711c06777" containerID="150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e" exitCode=0
Sep 17 18:59:48.828868 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:48.828461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerDied","Data":"150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e"}
Sep 17 18:59:50.818535 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:50.818504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 18:59:50.819036 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:50.819007    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:59:51.455757 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:51.455718    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:59:51.460097 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:51.460065    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:59:51.460490 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:51.460464    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:59:56.455616 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:56.455574    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:59:56.455984 ip-10-0-130-95 kubenswrapper[2565]: I0917 18:59:56.455771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 19:00:00.819981 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:00.819943    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:00:01.455613 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:01.455571    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 19:00:01.459919 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:01.459889    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:00:01.460244 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:01.460221    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:00:06.456017 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:06.455973    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 19:00:10.819301 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:10.819262    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:00:11.455711 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:11.455678    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 19:00:11.459968 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:11.459938    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:00:11.460082 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:11.460064    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 19:00:11.460290 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:11.460261    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:00:11.460404 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:11.460389    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 19:00:13.881553 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.881530    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 19:00:13.923445 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.923412    2565 generic.go:356] "Generic (PLEG): container finished" podID="a58b20c8-553f-445e-9dc0-917711c06777" containerID="0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc" exitCode=0
Sep 17 19:00:13.923576 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.923474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerDied","Data":"0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc"}
Sep 17 19:00:13.923576 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.923491    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"
Sep 17 19:00:13.923576 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.923541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd" event={"ID":"a58b20c8-553f-445e-9dc0-917711c06777","Type":"ContainerDied","Data":"77409085c8e7ad776db4a191a2103b85db6833727a89344e5661fcf45d1108ea"}
Sep 17 19:00:13.923576 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.923575    2565 scope.go:117] "RemoveContainer" containerID="0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc"
Sep 17 19:00:13.931265 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.931250    2565 scope.go:117] "RemoveContainer" containerID="86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b"
Sep 17 19:00:13.933333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-286ms\" (UniqueName: \"kubernetes.io/projected/a58b20c8-553f-445e-9dc0-917711c06777-kube-api-access-286ms\") pod \"a58b20c8-553f-445e-9dc0-917711c06777\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") "
Sep 17 19:00:13.933423 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls\") pod \"a58b20c8-553f-445e-9dc0-917711c06777\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") "
Sep 17 19:00:13.933423 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933407    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-kube-rbac-proxy-sar-config\") pod \"a58b20c8-553f-445e-9dc0-917711c06777\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") "
Sep 17 19:00:13.933534 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933491    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-cabundle-cert\") pod \"a58b20c8-553f-445e-9dc0-917711c06777\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") "
Sep 17 19:00:13.933534 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933529    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58b20c8-553f-445e-9dc0-917711c06777-kserve-provision-location\") pod \"a58b20c8-553f-445e-9dc0-917711c06777\" (UID: \"a58b20c8-553f-445e-9dc0-917711c06777\") "
Sep 17 19:00:13.933795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933760    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a58b20c8-553f-445e-9dc0-917711c06777" (UID: "a58b20c8-553f-445e-9dc0-917711c06777"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:00:13.933946 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933830    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:00:13.933946 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.933890    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a58b20c8-553f-445e-9dc0-917711c06777" (UID: "a58b20c8-553f-445e-9dc0-917711c06777"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:00:13.934069 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.934012    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58b20c8-553f-445e-9dc0-917711c06777-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a58b20c8-553f-445e-9dc0-917711c06777" (UID: "a58b20c8-553f-445e-9dc0-917711c06777"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:13.935582 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.935558    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58b20c8-553f-445e-9dc0-917711c06777-kube-api-access-286ms" (OuterVolumeSpecName: "kube-api-access-286ms") pod "a58b20c8-553f-445e-9dc0-917711c06777" (UID: "a58b20c8-553f-445e-9dc0-917711c06777"). InnerVolumeSpecName "kube-api-access-286ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:13.935876 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.935851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a58b20c8-553f-445e-9dc0-917711c06777" (UID: "a58b20c8-553f-445e-9dc0-917711c06777"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:13.938584 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.938563    2565 scope.go:117] "RemoveContainer" containerID="150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e"
Sep 17 19:00:13.949164 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.949071    2565 scope.go:117] "RemoveContainer" containerID="690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa"
Sep 17 19:00:13.955626 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.955610    2565 scope.go:117] "RemoveContainer" containerID="0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc"
Sep 17 19:00:13.955889 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:00:13.955871    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc\": container with ID starting with 0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc not found: ID does not exist" containerID="0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc"
Sep 17 19:00:13.955951 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.955895    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc"} err="failed to get container status \"0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc\": rpc error: code = NotFound desc = could not find container \"0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc\": container with ID starting with 0678b40e14e30bc0576a82f0bfbab6afbcfcce90abeabdb9cfc18fd8286e67cc not found: ID does not exist"
Sep 17 19:00:13.955951 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.955912    2565 scope.go:117] "RemoveContainer" containerID="86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b"
Sep 17 19:00:13.956161 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:00:13.956136    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b\": container with ID starting with 86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b not found: ID does not exist" containerID="86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b"
Sep 17 19:00:13.956200 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.956167    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b"} err="failed to get container status \"86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b\": rpc error: code = NotFound desc = could not find container \"86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b\": container with ID starting with 86648a277725dd38ba5787e2085978c213445b4af655d3feff4290c0f8c5e01b not found: ID does not exist"
Sep 17 19:00:13.956200 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.956185    2565 scope.go:117] "RemoveContainer" containerID="150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e"
Sep 17 19:00:13.956408 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:00:13.956393    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e\": container with ID starting with 150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e not found: ID does not exist" containerID="150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e"
Sep 17 19:00:13.956504 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.956410    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e"} err="failed to get container status \"150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e\": rpc error: code = NotFound desc = could not find container \"150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e\": container with ID starting with 150ef560bbf07983cd21760c3d3982212eb41cc2d90618cb59fe6e13036d541e not found: ID does not exist"
Sep 17 19:00:13.956504 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.956422    2565 scope.go:117] "RemoveContainer" containerID="690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa"
Sep 17 19:00:13.956695 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:00:13.956678    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa\": container with ID starting with 690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa not found: ID does not exist" containerID="690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa"
Sep 17 19:00:13.956737 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:13.956700    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa"} err="failed to get container status \"690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa\": rpc error: code = NotFound desc = could not find container \"690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa\": container with ID starting with 690dfa1aff3e48c7895c233585566b8394dafeb2a7716512363d75fb844c34fa not found: ID does not exist"
Sep 17 19:00:14.034160 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:14.034106    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58b20c8-553f-445e-9dc0-917711c06777-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:00:14.034160 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:14.034127    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58b20c8-553f-445e-9dc0-917711c06777-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:00:14.034160 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:14.034138    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286ms\" (UniqueName: \"kubernetes.io/projected/a58b20c8-553f-445e-9dc0-917711c06777-kube-api-access-286ms\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:00:14.034160 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:14.034147    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58b20c8-553f-445e-9dc0-917711c06777-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:00:14.246739 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:14.246710    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"]
Sep 17 19:00:14.250295 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:14.250274    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47a1c-predictor-685d4fb9b8-hzrbd"]
Sep 17 19:00:15.062265 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:15.062230    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58b20c8-553f-445e-9dc0-917711c06777" path="/var/lib/kubelet/pods/a58b20c8-553f-445e-9dc0-917711c06777/volumes"
Sep 17 19:00:20.819232 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:20.819188    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:00:30.819335 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:30.819293    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:00:40.819112 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:40.819069    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:00:50.819474 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:00:50.819374    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:01:00.819853 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:00.819823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 19:01:08.088253 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:08.088220    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"]
Sep 17 19:01:08.088669 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:08.088475    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" containerID="cri-o://4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79" gracePeriod=30
Sep 17 19:01:08.088669 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:08.088508    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kube-rbac-proxy" containerID="cri-o://f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863" gracePeriod=30
Sep 17 19:01:09.130533 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:09.130501    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad6d82cc-a179-4695-a910-705de69a1559" containerID="f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863" exitCode=2
Sep 17 19:01:09.130897 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:09.130575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerDied","Data":"f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863"}
Sep 17 19:01:10.814943 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:10.814894    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 19:01:10.819637 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:10.819614    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 19:01:12.130296 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.130269    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 19:01:12.142650 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.142628    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad6d82cc-a179-4695-a910-705de69a1559" containerID="4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79" exitCode=0
Sep 17 19:01:12.142752 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.142659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerDied","Data":"4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79"}
Sep 17 19:01:12.142752 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.142676    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"
Sep 17 19:01:12.142752 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.142710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8" event={"ID":"ad6d82cc-a179-4695-a910-705de69a1559","Type":"ContainerDied","Data":"0647758bfffbf367b29ddddbc7e8353916798b97950e226f032d69d39d4e5325"}
Sep 17 19:01:12.142752 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.142739    2565 scope.go:117] "RemoveContainer" containerID="f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863"
Sep 17 19:01:12.150522 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.150502    2565 scope.go:117] "RemoveContainer" containerID="4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79"
Sep 17 19:01:12.158265 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.158244    2565 scope.go:117] "RemoveContainer" containerID="48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9"
Sep 17 19:01:12.165465 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.165430    2565 scope.go:117] "RemoveContainer" containerID="f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863"
Sep 17 19:01:12.165755 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:01:12.165738    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863\": container with ID starting with f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863 not found: ID does not exist" containerID="f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863"
Sep 17 19:01:12.165809 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.165763    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863"} err="failed to get container status \"f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863\": rpc error: code = NotFound desc = could not find container \"f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863\": container with ID starting with f5aac2603213e8f06d162b7e83f65e846436a26bddc96bbd1c45554fdaa02863 not found: ID does not exist"
Sep 17 19:01:12.165809 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.165788    2565 scope.go:117] "RemoveContainer" containerID="4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79"
Sep 17 19:01:12.166038 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:01:12.166015    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79\": container with ID starting with 4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79 not found: ID does not exist" containerID="4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79"
Sep 17 19:01:12.166120 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.166048    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79"} err="failed to get container status \"4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79\": rpc error: code = NotFound desc = could not find container \"4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79\": container with ID starting with 4362fb8d58fd6610632ab039942def2faba66a79c60afc09157149eb38143e79 not found: ID does not exist"
Sep 17 19:01:12.166120 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.166069    2565 scope.go:117] "RemoveContainer" containerID="48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9"
Sep 17 19:01:12.166329 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:01:12.166312    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9\": container with ID starting with 48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9 not found: ID does not exist" containerID="48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9"
Sep 17 19:01:12.166368 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.166333    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9"} err="failed to get container status \"48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9\": rpc error: code = NotFound desc = could not find container \"48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9\": container with ID starting with 48ac40103d1a3a9a08c27245afa686a78471b982cd0c9b620f526d9437edf1e9 not found: ID does not exist"
Sep 17 19:01:12.231788 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.231736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnf69\" (UniqueName: \"kubernetes.io/projected/ad6d82cc-a179-4695-a910-705de69a1559-kube-api-access-cnf69\") pod \"ad6d82cc-a179-4695-a910-705de69a1559\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") "
Sep 17 19:01:12.231788 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.231768    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad6d82cc-a179-4695-a910-705de69a1559-kserve-provision-location\") pod \"ad6d82cc-a179-4695-a910-705de69a1559\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") "
Sep 17 19:01:12.231937 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.231793    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-cabundle-cert\") pod \"ad6d82cc-a179-4695-a910-705de69a1559\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") "
Sep 17 19:01:12.231937 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.231816    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-kube-rbac-proxy-sar-config\") pod \"ad6d82cc-a179-4695-a910-705de69a1559\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") "
Sep 17 19:01:12.231937 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.231842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad6d82cc-a179-4695-a910-705de69a1559-proxy-tls\") pod \"ad6d82cc-a179-4695-a910-705de69a1559\" (UID: \"ad6d82cc-a179-4695-a910-705de69a1559\") "
Sep 17 19:01:12.232116 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.232097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6d82cc-a179-4695-a910-705de69a1559-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad6d82cc-a179-4695-a910-705de69a1559" (UID: "ad6d82cc-a179-4695-a910-705de69a1559"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:12.232178 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.232144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad6d82cc-a179-4695-a910-705de69a1559" (UID: "ad6d82cc-a179-4695-a910-705de69a1559"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:12.232217 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.232148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad6d82cc-a179-4695-a910-705de69a1559" (UID: "ad6d82cc-a179-4695-a910-705de69a1559"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:12.233857 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.233826    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6d82cc-a179-4695-a910-705de69a1559-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad6d82cc-a179-4695-a910-705de69a1559" (UID: "ad6d82cc-a179-4695-a910-705de69a1559"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:12.233857 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.233835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6d82cc-a179-4695-a910-705de69a1559-kube-api-access-cnf69" (OuterVolumeSpecName: "kube-api-access-cnf69") pod "ad6d82cc-a179-4695-a910-705de69a1559" (UID: "ad6d82cc-a179-4695-a910-705de69a1559"). InnerVolumeSpecName "kube-api-access-cnf69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:12.332767 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.332737    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnf69\" (UniqueName: \"kubernetes.io/projected/ad6d82cc-a179-4695-a910-705de69a1559-kube-api-access-cnf69\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:01:12.332767 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.332770    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad6d82cc-a179-4695-a910-705de69a1559-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:01:12.332894 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.332781    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:01:12.332894 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.332791    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad6d82cc-a179-4695-a910-705de69a1559-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:01:12.332894 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.332800    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad6d82cc-a179-4695-a910-705de69a1559-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:01:12.467464 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.467423    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"]
Sep 17 19:01:12.470569 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:12.470547    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1cac1-predictor-598495cdf-kxwc8"]
Sep 17 19:01:13.063199 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.063159    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad6d82cc-a179-4695-a910-705de69a1559" path="/var/lib/kubelet/pods/ad6d82cc-a179-4695-a910-705de69a1559/volumes"
Sep 17 19:01:13.262258 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262222    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"]
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262498    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="storage-initializer"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262510    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="storage-initializer"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262521    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262526    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262532    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kube-rbac-proxy"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262538    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kube-rbac-proxy"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262546    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262551    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262563    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262568    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262575    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262580    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262588    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="storage-initializer"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262593    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="storage-initializer"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262635    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kube-rbac-proxy"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262643    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kserve-container"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262648    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="kserve-container"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262653    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58b20c8-553f-445e-9dc0-917711c06777" containerName="agent"
Sep 17 19:01:13.262666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.262658    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad6d82cc-a179-4695-a910-705de69a1559" containerName="kube-rbac-proxy"
Sep 17 19:01:13.267217 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.267202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.270362 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.270332    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:01:13.270362 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.270332    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-0de9c-kube-rbac-proxy-sar-config\""
Sep 17 19:01:13.270362 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.270363    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:01:13.270628 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.270334    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-0de9c-predictor-serving-cert\""
Sep 17 19:01:13.270704 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.270690    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:01:13.271344 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.271327    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 19:01:13.271424 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.271340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:01:13.278988 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.278965    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"]
Sep 17 19:01:13.339485 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.339411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kserve-provision-location\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.339485 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.339460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.339625 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.339522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2lg\" (UniqueName: \"kubernetes.io/projected/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-api-access-8h2lg\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.339625 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.339554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-cabundle-cert\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.339625 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.339584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-proxy-tls\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.440461 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.440421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2lg\" (UniqueName: \"kubernetes.io/projected/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-api-access-8h2lg\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.440598 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.440466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-cabundle-cert\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.440598 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.440491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-proxy-tls\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.440598 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.440533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kserve-provision-location\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.440598 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.440568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.440929 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.440912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kserve-provision-location\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.441217 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.441195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.441288 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.441195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-cabundle-cert\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.443033 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.443010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-proxy-tls\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.450716 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.450694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2lg\" (UniqueName: \"kubernetes.io/projected/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-api-access-8h2lg\") pod \"raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.576483 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.576461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:13.850109 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:13.850077    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"]
Sep 17 19:01:13.853067 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:01:13.853031    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c873a64_1f0d_45c9_9dba_9a80b6bc873d.slice/crio-b749c92a8f0656917eb9e2e9c68a24e92d326293fa6f6f90cf287a26a3d5ba16 WatchSource:0}: Error finding container b749c92a8f0656917eb9e2e9c68a24e92d326293fa6f6f90cf287a26a3d5ba16: Status 404 returned error can't find the container with id b749c92a8f0656917eb9e2e9c68a24e92d326293fa6f6f90cf287a26a3d5ba16
Sep 17 19:01:14.152675 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:14.152587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerStarted","Data":"753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b"}
Sep 17 19:01:14.152675 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:14.152644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerStarted","Data":"b749c92a8f0656917eb9e2e9c68a24e92d326293fa6f6f90cf287a26a3d5ba16"}
Sep 17 19:01:18.168106 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:18.168074    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerID="753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b" exitCode=0
Sep 17 19:01:18.168503 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:18.168148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerDied","Data":"753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b"}
Sep 17 19:01:19.173299 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:19.173264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerStarted","Data":"99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8"}
Sep 17 19:01:19.173671 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:19.173320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerStarted","Data":"e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566"}
Sep 17 19:01:19.173671 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:19.173627    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:19.173759 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:19.173674    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:19.174887 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:19.174860    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:01:20.176811 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:20.176767    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:01:25.182943 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:25.182914    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:01:25.183396 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:25.183372    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:01:25.207345 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:25.207295    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podStartSLOduration=8.483098814 podStartE2EDuration="12.207279868s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.724181054s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:19.201960339 +0000 UTC m=+1368.488272856" watchObservedRunningTime="2026-09-17 19:01:25.207279868 +0000 UTC m=+1374.493592384"
Sep 17 19:01:35.183543 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:35.183504    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:01:45.183857 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:45.183816    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:01:55.184264 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:01:55.184226    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:02:05.183673 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:05.183635    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:02:15.183810 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:15.183731    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:02:25.183556 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:25.183518    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:02:29.155037 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.154994    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"]
Sep 17 19:02:29.159714 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.159692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.162330 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.162313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 19:02:29.162421 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.162333    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 19:02:29.168494 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.168472    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"]
Sep 17 19:02:29.258355 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.258326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e46642-1628-494b-ae65-24b291aa2478-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.258528 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.258377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.258528 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.258407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.258528 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.258486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8zp\" (UniqueName: \"kubernetes.io/projected/a3e46642-1628-494b-ae65-24b291aa2478-kube-api-access-vf8zp\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.258528 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.258525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.285145 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.285113    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"]
Sep 17 19:02:29.288151 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.288136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.290853 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.290831    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 19:02:29.306780 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.306751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"]
Sep 17 19:02:29.359131 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359263 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsvw\" (UniqueName: \"kubernetes.io/projected/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-api-access-lgsvw\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.359263 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359263 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8zp\" (UniqueName: \"kubernetes.io/projected/a3e46642-1628-494b-ae65-24b291aa2478-kube-api-access-vf8zp\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359263 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.359470 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:02:29.359279    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 19:02:29.359470 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb978519-dc2e-41f4-a56e-ae17ba4073e3-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.359470 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359470 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:02:29.359346    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls podName:a3e46642-1628-494b-ae65-24b291aa2478 nodeName:}" failed. No retries permitted until 2026-09-17 19:02:29.859329421 +0000 UTC m=+1439.145641919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls") pod "isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" (UID: "a3e46642-1628-494b-ae65-24b291aa2478") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 19:02:29.359470 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.359470 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.359775 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e46642-1628-494b-ae65-24b291aa2478-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359838 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e46642-1628-494b-ae65-24b291aa2478-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359950 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.359988 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.359932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.372996 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.372977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8zp\" (UniqueName: \"kubernetes.io/projected/a3e46642-1628-494b-ae65-24b291aa2478-kube-api-access-vf8zp\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.460769 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.460737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsvw\" (UniqueName: \"kubernetes.io/projected/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-api-access-lgsvw\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.460874 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.460790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.460924 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.460899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb978519-dc2e-41f4-a56e-ae17ba4073e3-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.460976 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.460930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.460976 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.460948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.461096 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.461057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.461675 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.461654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.461751 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.461654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.463219 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.463202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb978519-dc2e-41f4-a56e-ae17ba4073e3-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.475314 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.475296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsvw\" (UniqueName: \"kubernetes.io/projected/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-api-access-lgsvw\") pod \"isvc-canary-create-predictor-b75cfc78d-nbs77\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.598190 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.598168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:29.864752 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.864669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.866919 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.866902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:29.876662 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:29.876642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"]
Sep 17 19:02:29.879059 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:02:29.879037    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb978519_dc2e_41f4_a56e_ae17ba4073e3.slice/crio-a76c29cee8d693b2e6a76edaf77788aeedff441a15ed46115eb0b7b2d5613712 WatchSource:0}: Error finding container a76c29cee8d693b2e6a76edaf77788aeedff441a15ed46115eb0b7b2d5613712: Status 404 returned error can't find the container with id a76c29cee8d693b2e6a76edaf77788aeedff441a15ed46115eb0b7b2d5613712
Sep 17 19:02:30.070559 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:30.070529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:30.353664 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:30.353532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"]
Sep 17 19:02:30.355981 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:02:30.355949    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e46642_1628_494b_ae65_24b291aa2478.slice/crio-2ec04bf2703a3daf09f6f63ee8d284280f7cfe61b5a356dee8a9133c1ea1cca4 WatchSource:0}: Error finding container 2ec04bf2703a3daf09f6f63ee8d284280f7cfe61b5a356dee8a9133c1ea1cca4: Status 404 returned error can't find the container with id 2ec04bf2703a3daf09f6f63ee8d284280f7cfe61b5a356dee8a9133c1ea1cca4
Sep 17 19:02:30.430597 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:30.430556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerStarted","Data":"335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee"}
Sep 17 19:02:30.430713 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:30.430635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerStarted","Data":"a76c29cee8d693b2e6a76edaf77788aeedff441a15ed46115eb0b7b2d5613712"}
Sep 17 19:02:30.432190 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:30.432162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerStarted","Data":"c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e"}
Sep 17 19:02:30.432288 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:30.432224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerStarted","Data":"2ec04bf2703a3daf09f6f63ee8d284280f7cfe61b5a356dee8a9133c1ea1cca4"}
Sep 17 19:02:34.452185 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:34.452147    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerID="335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee" exitCode=0
Sep 17 19:02:34.452628 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:34.452227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerDied","Data":"335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee"}
Sep 17 19:02:34.453983 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:34.453964    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3e46642-1628-494b-ae65-24b291aa2478" containerID="c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e" exitCode=0
Sep 17 19:02:34.454086 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:34.453996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerDied","Data":"c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e"}
Sep 17 19:02:35.062452 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.062416    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:02:35.460462 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.460412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerStarted","Data":"eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d"}
Sep 17 19:02:35.460914 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.460494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerStarted","Data":"36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad"}
Sep 17 19:02:35.460914 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.460678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:35.462484 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.462462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerStarted","Data":"d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947"}
Sep 17 19:02:35.462593 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.462511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerStarted","Data":"84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc"}
Sep 17 19:02:35.462841 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.462826    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:35.484684 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.484625    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podStartSLOduration=2.653519513 podStartE2EDuration="6.484611144s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831091631s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:35.483689338 +0000 UTC m=+1444.770001855" watchObservedRunningTime="2026-09-17 19:02:35.484611144 +0000 UTC m=+1444.770923722"
Sep 17 19:02:35.506728 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:35.506679    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podStartSLOduration=2.768364515 podStartE2EDuration="6.506664535s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.73830002s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:35.503429905 +0000 UTC m=+1444.789742422" watchObservedRunningTime="2026-09-17 19:02:35.506664535 +0000 UTC m=+1444.792977053"
Sep 17 19:02:36.466488 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:36.466456    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:36.466991 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:36.466506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:36.467981 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:36.467958    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:02:36.468068 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:36.468042    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:02:37.469731 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:37.469687    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:02:37.470214 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:37.469784    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:02:42.473907 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:42.473880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:02:42.474377 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:42.474320    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:02:42.474464 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:42.474313    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:02:42.474814 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:42.474789    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:02:43.317481 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:43.317425    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"]
Sep 17 19:02:43.317726 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:43.317705    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" containerID="cri-o://e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566" gracePeriod=30
Sep 17 19:02:43.317795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:43.317746    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kube-rbac-proxy" containerID="cri-o://99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8" gracePeriod=30
Sep 17 19:02:43.500170 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:43.500139    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerID="99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8" exitCode=2
Sep 17 19:02:43.500563 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:43.500208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerDied","Data":"99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8"}
Sep 17 19:02:45.057952 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:45.057909    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:02:45.177605 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:45.177558    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 19:02:47.963197 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:47.963175    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:02:48.108988 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.108916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h2lg\" (UniqueName: \"kubernetes.io/projected/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-api-access-8h2lg\") pod \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") "
Sep 17 19:02:48.108988 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.108962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-rbac-proxy-sar-config\") pod \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") "
Sep 17 19:02:48.109191 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.109001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kserve-provision-location\") pod \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") "
Sep 17 19:02:48.109191 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.109018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-cabundle-cert\") pod \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") "
Sep 17 19:02:48.109191 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.109063    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-proxy-tls\") pod \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\" (UID: \"1c873a64-1f0d-45c9-9dba-9a80b6bc873d\") "
Sep 17 19:02:48.109368 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.109338    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c873a64-1f0d-45c9-9dba-9a80b6bc873d" (UID: "1c873a64-1f0d-45c9-9dba-9a80b6bc873d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:48.109455 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.109411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c873a64-1f0d-45c9-9dba-9a80b6bc873d" (UID: "1c873a64-1f0d-45c9-9dba-9a80b6bc873d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:48.109523 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.109479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c873a64-1f0d-45c9-9dba-9a80b6bc873d" (UID: "1c873a64-1f0d-45c9-9dba-9a80b6bc873d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:48.111086 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.111058    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c873a64-1f0d-45c9-9dba-9a80b6bc873d" (UID: "1c873a64-1f0d-45c9-9dba-9a80b6bc873d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:48.111140 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.111106    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-api-access-8h2lg" (OuterVolumeSpecName: "kube-api-access-8h2lg") pod "1c873a64-1f0d-45c9-9dba-9a80b6bc873d" (UID: "1c873a64-1f0d-45c9-9dba-9a80b6bc873d"). InnerVolumeSpecName "kube-api-access-8h2lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:48.210068 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.210033    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.210068 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.210065    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h2lg\" (UniqueName: \"kubernetes.io/projected/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-api-access-8h2lg\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.210068 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.210075    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.210068 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.210084    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.210068 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.210094    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c873a64-1f0d-45c9-9dba-9a80b6bc873d-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.521625 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.521594    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerID="e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566" exitCode=0
Sep 17 19:02:48.521815 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.521648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerDied","Data":"e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566"}
Sep 17 19:02:48.521815 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.521658    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"
Sep 17 19:02:48.521815 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.521708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb" event={"ID":"1c873a64-1f0d-45c9-9dba-9a80b6bc873d","Type":"ContainerDied","Data":"b749c92a8f0656917eb9e2e9c68a24e92d326293fa6f6f90cf287a26a3d5ba16"}
Sep 17 19:02:48.521815 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.521736    2565 scope.go:117] "RemoveContainer" containerID="99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8"
Sep 17 19:02:48.530733 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.530718    2565 scope.go:117] "RemoveContainer" containerID="e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566"
Sep 17 19:02:48.537901 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.537885    2565 scope.go:117] "RemoveContainer" containerID="753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b"
Sep 17 19:02:48.544393 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.544373    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"]
Sep 17 19:02:48.545311 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.545296    2565 scope.go:117] "RemoveContainer" containerID="99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8"
Sep 17 19:02:48.545636 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:02:48.545612    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8\": container with ID starting with 99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8 not found: ID does not exist" containerID="99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8"
Sep 17 19:02:48.545733 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.545640    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8"} err="failed to get container status \"99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8\": rpc error: code = NotFound desc = could not find container \"99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8\": container with ID starting with 99c5dab790bd2861126f269cf35afcc274d6c38b01d5d625eb40c02b8c93bff8 not found: ID does not exist"
Sep 17 19:02:48.545733 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.545656    2565 scope.go:117] "RemoveContainer" containerID="e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566"
Sep 17 19:02:48.545953 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:02:48.545939    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566\": container with ID starting with e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566 not found: ID does not exist" containerID="e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566"
Sep 17 19:02:48.546007 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.545956    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566"} err="failed to get container status \"e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566\": rpc error: code = NotFound desc = could not find container \"e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566\": container with ID starting with e81cdfc48f6708f1e78301f49fa35c80e10710dc219901a4bb33b0a4a36a4566 not found: ID does not exist"
Sep 17 19:02:48.546007 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.545969    2565 scope.go:117] "RemoveContainer" containerID="753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b"
Sep 17 19:02:48.546912 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:02:48.546885    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b\": container with ID starting with 753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b not found: ID does not exist" containerID="753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b"
Sep 17 19:02:48.547015 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.546935    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b"} err="failed to get container status \"753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b\": rpc error: code = NotFound desc = could not find container \"753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b\": container with ID starting with 753df02088f50ef5de7806850794934d18167cac48ef7eaea41efbf6f4d7500b not found: ID does not exist"
Sep 17 19:02:48.548450 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:48.548411    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0de9c-predictor-74978bb977-jw7hb"]
Sep 17 19:02:49.062097 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:49.062066    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" path="/var/lib/kubelet/pods/1c873a64-1f0d-45c9-9dba-9a80b6bc873d/volumes"
Sep 17 19:02:52.474351 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:52.474316    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:02:52.474874 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:02:52.474772    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:03:02.474896 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:02.474856    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:03:02.475348 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:02.474857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:03:12.475005 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:12.474968    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:03:12.475400 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:12.474965    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:03:22.474431 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:22.474385    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:03:22.474853 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:22.474650    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:03:32.475259 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:32.475220    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:03:32.475750 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:32.475221    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:03:42.474604 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:42.474562    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:03:42.475077 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:42.474714    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:03:45.058006 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:45.057914    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:03:52.475645 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.475609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:03:52.616312 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.616279    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"]
Sep 17 19:03:52.616585 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.616540    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container" containerID="cri-o://84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc" gracePeriod=30
Sep 17 19:03:52.616687 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.616566    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kube-rbac-proxy" containerID="cri-o://d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947" gracePeriod=30
Sep 17 19:03:52.617385 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.617171    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:03:52.774936 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.774903    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerID="d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947" exitCode=2
Sep 17 19:03:52.775074 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:52.774960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerDied","Data":"d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947"}
Sep 17 19:03:56.954108 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:56.954083    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:03:57.028141 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-cabundle-cert\") pod \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") "
Sep 17 19:03:57.028141 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgsvw\" (UniqueName: \"kubernetes.io/projected/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-api-access-lgsvw\") pod \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") "
Sep 17 19:03:57.028141 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kserve-provision-location\") pod \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") "
Sep 17 19:03:57.028384 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028197    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb978519-dc2e-41f4-a56e-ae17ba4073e3-proxy-tls\") pod \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") "
Sep 17 19:03:57.028384 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-rbac-proxy-sar-config\") pod \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\" (UID: \"eb978519-dc2e-41f4-a56e-ae17ba4073e3\") "
Sep 17 19:03:57.028532 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb978519-dc2e-41f4-a56e-ae17ba4073e3" (UID: "eb978519-dc2e-41f4-a56e-ae17ba4073e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:57.028588 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb978519-dc2e-41f4-a56e-ae17ba4073e3" (UID: "eb978519-dc2e-41f4-a56e-ae17ba4073e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:57.028736 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.028710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb978519-dc2e-41f4-a56e-ae17ba4073e3" (UID: "eb978519-dc2e-41f4-a56e-ae17ba4073e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:57.030120 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.030095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-api-access-lgsvw" (OuterVolumeSpecName: "kube-api-access-lgsvw") pod "eb978519-dc2e-41f4-a56e-ae17ba4073e3" (UID: "eb978519-dc2e-41f4-a56e-ae17ba4073e3"). InnerVolumeSpecName "kube-api-access-lgsvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:57.030207 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.030146    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb978519-dc2e-41f4-a56e-ae17ba4073e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb978519-dc2e-41f4-a56e-ae17ba4073e3" (UID: "eb978519-dc2e-41f4-a56e-ae17ba4073e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:57.128788 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.128765    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:03:57.128788 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.128786    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb978519-dc2e-41f4-a56e-ae17ba4073e3-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:03:57.128923 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.128794    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgsvw\" (UniqueName: \"kubernetes.io/projected/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kube-api-access-lgsvw\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:03:57.128923 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.128803    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb978519-dc2e-41f4-a56e-ae17ba4073e3-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:03:57.128923 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.128811    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb978519-dc2e-41f4-a56e-ae17ba4073e3-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:03:57.795577 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.795538    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerID="84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc" exitCode=0
Sep 17 19:03:57.795750 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.795603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"
Sep 17 19:03:57.795750 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.795621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerDied","Data":"84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc"}
Sep 17 19:03:57.795750 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.795696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77" event={"ID":"eb978519-dc2e-41f4-a56e-ae17ba4073e3","Type":"ContainerDied","Data":"a76c29cee8d693b2e6a76edaf77788aeedff441a15ed46115eb0b7b2d5613712"}
Sep 17 19:03:57.795750 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.795732    2565 scope.go:117] "RemoveContainer" containerID="d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947"
Sep 17 19:03:57.803635 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.803621    2565 scope.go:117] "RemoveContainer" containerID="84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc"
Sep 17 19:03:57.810595 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.810577    2565 scope.go:117] "RemoveContainer" containerID="335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee"
Sep 17 19:03:57.814419 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.814397    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"]
Sep 17 19:03:57.817532 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.817511    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-nbs77"]
Sep 17 19:03:57.818305 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.818289    2565 scope.go:117] "RemoveContainer" containerID="d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947"
Sep 17 19:03:57.818669 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:03:57.818652    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947\": container with ID starting with d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947 not found: ID does not exist" containerID="d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947"
Sep 17 19:03:57.818744 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.818676    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947"} err="failed to get container status \"d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947\": rpc error: code = NotFound desc = could not find container \"d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947\": container with ID starting with d014e70545ae0cc5df3f668059bde80a0b95d33920f2e8a95a21f666112d3947 not found: ID does not exist"
Sep 17 19:03:57.818744 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.818693    2565 scope.go:117] "RemoveContainer" containerID="84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc"
Sep 17 19:03:57.818949 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:03:57.818931    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc\": container with ID starting with 84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc not found: ID does not exist" containerID="84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc"
Sep 17 19:03:57.818987 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.818956    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc"} err="failed to get container status \"84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc\": rpc error: code = NotFound desc = could not find container \"84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc\": container with ID starting with 84026b61187faa18e592abad8138c8d840e440db800ebdc0c8ad68028907dabc not found: ID does not exist"
Sep 17 19:03:57.818987 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.818970    2565 scope.go:117] "RemoveContainer" containerID="335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee"
Sep 17 19:03:57.819208 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:03:57.819190    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee\": container with ID starting with 335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee not found: ID does not exist" containerID="335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee"
Sep 17 19:03:57.819262 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:57.819214    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee"} err="failed to get container status \"335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee\": rpc error: code = NotFound desc = could not find container \"335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee\": container with ID starting with 335946a02aa424bc9857725b9260ea347683b578ffa91c9a920686c6a662a7ee not found: ID does not exist"
Sep 17 19:03:59.061946 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:59.061919    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" path="/var/lib/kubelet/pods/eb978519-dc2e-41f4-a56e-ae17ba4073e3/volumes"
Sep 17 19:03:59.250290 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:59.250261    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"]
Sep 17 19:03:59.250528 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:59.250503    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" containerID="cri-o://36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad" gracePeriod=30
Sep 17 19:03:59.250631 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:59.250546    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kube-rbac-proxy" containerID="cri-o://eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d" gracePeriod=30
Sep 17 19:03:59.806695 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:59.806662    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3e46642-1628-494b-ae65-24b291aa2478" containerID="eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d" exitCode=2
Sep 17 19:03:59.806861 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:03:59.806735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerDied","Data":"eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d"}
Sep 17 19:04:02.470676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:02.470635    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 19:04:02.474929 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:02.474905    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:04:03.202088 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.202061    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:04:03.273358 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273319    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-cabundle-cert\") pod \"a3e46642-1628-494b-ae65-24b291aa2478\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") "
Sep 17 19:04:03.273619 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf8zp\" (UniqueName: \"kubernetes.io/projected/a3e46642-1628-494b-ae65-24b291aa2478-kube-api-access-vf8zp\") pod \"a3e46642-1628-494b-ae65-24b291aa2478\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") "
Sep 17 19:04:03.273619 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273482    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls\") pod \"a3e46642-1628-494b-ae65-24b291aa2478\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") "
Sep 17 19:04:03.273619 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273512    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e46642-1628-494b-ae65-24b291aa2478-kserve-provision-location\") pod \"a3e46642-1628-494b-ae65-24b291aa2478\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") "
Sep 17 19:04:03.273619 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273541    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-kube-rbac-proxy-sar-config\") pod \"a3e46642-1628-494b-ae65-24b291aa2478\" (UID: \"a3e46642-1628-494b-ae65-24b291aa2478\") "
Sep 17 19:04:03.273805 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3e46642-1628-494b-ae65-24b291aa2478" (UID: "a3e46642-1628-494b-ae65-24b291aa2478"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:03.273874 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273847    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e46642-1628-494b-ae65-24b291aa2478-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3e46642-1628-494b-ae65-24b291aa2478" (UID: "a3e46642-1628-494b-ae65-24b291aa2478"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:03.273960 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.273938    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3e46642-1628-494b-ae65-24b291aa2478" (UID: "a3e46642-1628-494b-ae65-24b291aa2478"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:03.275459 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.275425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3e46642-1628-494b-ae65-24b291aa2478" (UID: "a3e46642-1628-494b-ae65-24b291aa2478"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:03.275533 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.275492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e46642-1628-494b-ae65-24b291aa2478-kube-api-access-vf8zp" (OuterVolumeSpecName: "kube-api-access-vf8zp") pod "a3e46642-1628-494b-ae65-24b291aa2478" (UID: "a3e46642-1628-494b-ae65-24b291aa2478"). InnerVolumeSpecName "kube-api-access-vf8zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:03.374516 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.374388    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8zp\" (UniqueName: \"kubernetes.io/projected/a3e46642-1628-494b-ae65-24b291aa2478-kube-api-access-vf8zp\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.374516 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.374424    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e46642-1628-494b-ae65-24b291aa2478-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.374516 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.374465    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e46642-1628-494b-ae65-24b291aa2478-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.374516 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.374477    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.374516 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.374489    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e46642-1628-494b-ae65-24b291aa2478-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.822965 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.822932    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3e46642-1628-494b-ae65-24b291aa2478" containerID="36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad" exitCode=0
Sep 17 19:04:03.823367 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.822986    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"
Sep 17 19:04:03.823367 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.823000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerDied","Data":"36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad"}
Sep 17 19:04:03.823367 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.823050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp" event={"ID":"a3e46642-1628-494b-ae65-24b291aa2478","Type":"ContainerDied","Data":"2ec04bf2703a3daf09f6f63ee8d284280f7cfe61b5a356dee8a9133c1ea1cca4"}
Sep 17 19:04:03.823367 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.823077    2565 scope.go:117] "RemoveContainer" containerID="eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d"
Sep 17 19:04:03.831810 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.831786    2565 scope.go:117] "RemoveContainer" containerID="36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad"
Sep 17 19:04:03.839717 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.839695    2565 scope.go:117] "RemoveContainer" containerID="c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e"
Sep 17 19:04:03.848996 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.848968    2565 scope.go:117] "RemoveContainer" containerID="eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d"
Sep 17 19:04:03.849587 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:04:03.849561    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d\": container with ID starting with eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d not found: ID does not exist" containerID="eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d"
Sep 17 19:04:03.849709 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.849596    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d"} err="failed to get container status \"eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d\": rpc error: code = NotFound desc = could not find container \"eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d\": container with ID starting with eb740babd333acd8e6d9f018003efadb6b99a50c1ba46a933bd8610155e8b95d not found: ID does not exist"
Sep 17 19:04:03.849709 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.849616    2565 scope.go:117] "RemoveContainer" containerID="36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad"
Sep 17 19:04:03.849999 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:04:03.849975    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad\": container with ID starting with 36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad not found: ID does not exist" containerID="36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad"
Sep 17 19:04:03.850060 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.850006    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad"} err="failed to get container status \"36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad\": rpc error: code = NotFound desc = could not find container \"36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad\": container with ID starting with 36b55381d0dd3988cb40541191e0f8d40778c08360a6bd4f5e7af5da3ef8f9ad not found: ID does not exist"
Sep 17 19:04:03.850060 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.850022    2565 scope.go:117] "RemoveContainer" containerID="c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e"
Sep 17 19:04:03.850325 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:04:03.850309    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e\": container with ID starting with c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e not found: ID does not exist" containerID="c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e"
Sep 17 19:04:03.850367 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.850337    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e"} err="failed to get container status \"c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e\": rpc error: code = NotFound desc = could not find container \"c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e\": container with ID starting with c547bec611afa5d23ef81f7695b8b02767dfb59a86f3bba3be09a5f13d54b67e not found: ID does not exist"
Sep 17 19:04:03.854705 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.854673    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"]
Sep 17 19:04:03.872043 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:03.872007    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-tcpcp"]
Sep 17 19:04:04.744334 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744300    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"]
Sep 17 19:04:04.744625 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744611    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="storage-initializer"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744627    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="storage-initializer"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744636    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744642    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744652    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744660    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744667    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744672    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container"
Sep 17 19:04:04.744676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744678    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744683    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744693    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744698    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744706    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="storage-initializer"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744711    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="storage-initializer"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744717    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="storage-initializer"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="storage-initializer"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744729    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744733    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744778    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744784    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c873a64-1f0d-45c9-9dba-9a80b6bc873d" containerName="kserve-container"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744791    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kube-rbac-proxy"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744796    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e46642-1628-494b-ae65-24b291aa2478" containerName="kserve-container"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744801    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kserve-container"
Sep 17 19:04:04.744913 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.744806    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb978519-dc2e-41f4-a56e-ae17ba4073e3" containerName="kube-rbac-proxy"
Sep 17 19:04:04.749146 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.749128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.753581 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.753553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:04:04.754812 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.754786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 19:04:04.754989 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.754838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:04:04.754989 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.754843    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 19:04:04.754989 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.754874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:04:04.754989 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.754916    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:04:04.754989 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.754840    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:04:04.877222 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.877189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"]
Sep 17 19:04:04.886756 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.886726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.887015 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.886993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.887096 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.887028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.887096 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.887055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.887258 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.887161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6vs\" (UniqueName: \"kubernetes.io/projected/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-api-access-7l6vs\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.918835 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.918796    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"]
Sep 17 19:04:04.922150 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.922129    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:04.925786 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.925764    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 19:04:04.943031 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.943009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"]
Sep 17 19:04:04.988083 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988050    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f0932d7-d1f4-4897-92a6-6297122fdc37-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:04.988083 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6vs\" (UniqueName: \"kubernetes.io/projected/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-api-access-7l6vs\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.988354 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f0932d7-d1f4-4897-92a6-6297122fdc37-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:04.988354 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.988354 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgtf\" (UniqueName: \"kubernetes.io/projected/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-api-access-dcgtf\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:04.988354 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:04.988354 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.988618 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.988618 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.988618 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:04.988778 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.988621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.989058 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.989028    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.989201 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.989109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.990779 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.990757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:04.997632 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:04.997581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6vs\" (UniqueName: \"kubernetes.io/projected/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-api-access-7l6vs\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:05.058766 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.058729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:05.061333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.061305    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e46642-1628-494b-ae65-24b291aa2478" path="/var/lib/kubelet/pods/a3e46642-1628-494b-ae65-24b291aa2478/volumes"
Sep 17 19:04:05.088867 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.088839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089019 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.088879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f0932d7-d1f4-4897-92a6-6297122fdc37-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089019 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.088903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f0932d7-d1f4-4897-92a6-6297122fdc37-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089019 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.088933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgtf\" (UniqueName: \"kubernetes.io/projected/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-api-access-dcgtf\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089019 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.088972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089345 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.089323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f0932d7-d1f4-4897-92a6-6297122fdc37-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089576 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.089549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.089690 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.089604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.091405 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.091380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f0932d7-d1f4-4897-92a6-6297122fdc37-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.098860 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.098830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgtf\" (UniqueName: \"kubernetes.io/projected/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-api-access-dcgtf\") pod \"isvc-canary-promote-predictor-5f4f848c44-42kcb\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.231975 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.231931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:05.345452 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.345402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"]
Sep 17 19:04:05.350099 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:04:05.350075    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6e00ed_ab03_477c_b9ae_7cc4043c4f8d.slice/crio-f68a5857f4461bbf755b3d35982eb62f152fd21f4ceaa90f2449e0345f72a3cb WatchSource:0}: Error finding container f68a5857f4461bbf755b3d35982eb62f152fd21f4ceaa90f2449e0345f72a3cb: Status 404 returned error can't find the container with id f68a5857f4461bbf755b3d35982eb62f152fd21f4ceaa90f2449e0345f72a3cb
Sep 17 19:04:05.510478 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.510424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"]
Sep 17 19:04:05.513926 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:04:05.513890    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0932d7_d1f4_4897_92a6_6297122fdc37.slice/crio-5a0b50ef6d7a2189335c380d837b329a0d3bc325d6af7303b294d00ce9c8c10a WatchSource:0}: Error finding container 5a0b50ef6d7a2189335c380d837b329a0d3bc325d6af7303b294d00ce9c8c10a: Status 404 returned error can't find the container with id 5a0b50ef6d7a2189335c380d837b329a0d3bc325d6af7303b294d00ce9c8c10a
Sep 17 19:04:05.834218 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.834118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerStarted","Data":"5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2"}
Sep 17 19:04:05.834391 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.834231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerStarted","Data":"f68a5857f4461bbf755b3d35982eb62f152fd21f4ceaa90f2449e0345f72a3cb"}
Sep 17 19:04:05.835889 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.835862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerStarted","Data":"38392937282f3a9a42c8405052ac42c20783debe3e7b4588702089d71cf3fc3b"}
Sep 17 19:04:05.836028 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:05.835918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerStarted","Data":"5a0b50ef6d7a2189335c380d837b329a0d3bc325d6af7303b294d00ce9c8c10a"}
Sep 17 19:04:09.852032 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:09.852000    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerID="38392937282f3a9a42c8405052ac42c20783debe3e7b4588702089d71cf3fc3b" exitCode=0
Sep 17 19:04:09.852471 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:09.852073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerDied","Data":"38392937282f3a9a42c8405052ac42c20783debe3e7b4588702089d71cf3fc3b"}
Sep 17 19:04:09.853703 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:09.853679    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerID="5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2" exitCode=0
Sep 17 19:04:09.853796 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:09.853748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerDied","Data":"5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2"}
Sep 17 19:04:10.859710 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.859674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerStarted","Data":"974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97"}
Sep 17 19:04:10.860116 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.859741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerStarted","Data":"9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2"}
Sep 17 19:04:10.860116 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.859952    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:10.861759 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.861734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerStarted","Data":"65f30b14ed3113418edabdab9e3bef3fc2e8199e9d5193d3e62a7b2e72514175"}
Sep 17 19:04:10.861834 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.861786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerStarted","Data":"91d35e0993fa870845729a25dfcf8fbc15a0b8542faef96bc7a727b88349ea2d"}
Sep 17 19:04:10.861997 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.861982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:10.882909 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.882857    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podStartSLOduration=3.019270209 podStartE2EDuration="6.882840936s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.863570727s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:10.881330403 +0000 UTC m=+1540.167642919" watchObservedRunningTime="2026-09-17 19:04:10.882840936 +0000 UTC m=+1540.169153453"
Sep 17 19:04:10.902875 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:10.902819    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podStartSLOduration=3.033975345 podStartE2EDuration="6.902801805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.86882646s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:10.901472669 +0000 UTC m=+1540.187785186" watchObservedRunningTime="2026-09-17 19:04:10.902801805 +0000 UTC m=+1540.189114321"
Sep 17 19:04:11.866551 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:11.866415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:11.866551 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:11.866505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:11.867521 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:11.867493    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:11.867521 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:11.867507    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:04:12.870455 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:12.870394    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:04:12.870855 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:12.870547    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:17.875171 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:17.875140    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:04:17.875712 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:17.875689    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:04:17.875764 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:17.875736    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:17.876201 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:17.876180    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:04:27.875759 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:27.875721    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:27.878236 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:27.876155    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:04:37.876318 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:37.876277    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:37.876318 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:37.876277    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:04:47.876476 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:47.876421    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:47.876853 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:47.876421    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:04:57.876255 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:57.876216    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:04:57.876659 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:04:57.876227    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:05:07.876135 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:07.876101    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:05:07.876613 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:07.876099    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:05:17.875897 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:17.875809    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:05:17.876270 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:17.876122    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:05:27.876614 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:27.876583    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:05:27.877071 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:27.877035    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:05:34.605502 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:34.605463    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"]
Sep 17 19:05:34.605963 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:34.605768    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" containerID="cri-o://91d35e0993fa870845729a25dfcf8fbc15a0b8542faef96bc7a727b88349ea2d" gracePeriod=30
Sep 17 19:05:34.605963 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:34.605802    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kube-rbac-proxy" containerID="cri-o://65f30b14ed3113418edabdab9e3bef3fc2e8199e9d5193d3e62a7b2e72514175" gracePeriod=30
Sep 17 19:05:35.183282 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:35.183250    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerID="65f30b14ed3113418edabdab9e3bef3fc2e8199e9d5193d3e62a7b2e72514175" exitCode=2
Sep 17 19:05:35.183502 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:35.183322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerDied","Data":"65f30b14ed3113418edabdab9e3bef3fc2e8199e9d5193d3e62a7b2e72514175"}
Sep 17 19:05:37.870817 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:37.870767    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 19:05:37.876123 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:37.876094    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 19:05:39.201926 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.201896    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerID="91d35e0993fa870845729a25dfcf8fbc15a0b8542faef96bc7a727b88349ea2d" exitCode=0
Sep 17 19:05:39.202259 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.201962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerDied","Data":"91d35e0993fa870845729a25dfcf8fbc15a0b8542faef96bc7a727b88349ea2d"}
Sep 17 19:05:39.241387 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.241365    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:05:39.314035 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.313943    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-cabundle-cert\") pod \"7f0932d7-d1f4-4897-92a6-6297122fdc37\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") "
Sep 17 19:05:39.314035 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.313995    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f0932d7-d1f4-4897-92a6-6297122fdc37-proxy-tls\") pod \"7f0932d7-d1f4-4897-92a6-6297122fdc37\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") "
Sep 17 19:05:39.314035 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.314037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f0932d7-d1f4-4897-92a6-6297122fdc37-kserve-provision-location\") pod \"7f0932d7-d1f4-4897-92a6-6297122fdc37\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") "
Sep 17 19:05:39.314317 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.314067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcgtf\" (UniqueName: \"kubernetes.io/projected/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-api-access-dcgtf\") pod \"7f0932d7-d1f4-4897-92a6-6297122fdc37\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") "
Sep 17 19:05:39.314317 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.314092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-rbac-proxy-sar-config\") pod \"7f0932d7-d1f4-4897-92a6-6297122fdc37\" (UID: \"7f0932d7-d1f4-4897-92a6-6297122fdc37\") "
Sep 17 19:05:39.314422 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.314366    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0932d7-d1f4-4897-92a6-6297122fdc37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f0932d7-d1f4-4897-92a6-6297122fdc37" (UID: "7f0932d7-d1f4-4897-92a6-6297122fdc37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:39.314518 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.314479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f0932d7-d1f4-4897-92a6-6297122fdc37" (UID: "7f0932d7-d1f4-4897-92a6-6297122fdc37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:39.314518 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.314497    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f0932d7-d1f4-4897-92a6-6297122fdc37" (UID: "7f0932d7-d1f4-4897-92a6-6297122fdc37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:39.316202 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.316182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-api-access-dcgtf" (OuterVolumeSpecName: "kube-api-access-dcgtf") pod "7f0932d7-d1f4-4897-92a6-6297122fdc37" (UID: "7f0932d7-d1f4-4897-92a6-6297122fdc37"). InnerVolumeSpecName "kube-api-access-dcgtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:05:39.316254 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.316186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0932d7-d1f4-4897-92a6-6297122fdc37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f0932d7-d1f4-4897-92a6-6297122fdc37" (UID: "7f0932d7-d1f4-4897-92a6-6297122fdc37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:39.415005 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.414969    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:05:39.415005 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.414997    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f0932d7-d1f4-4897-92a6-6297122fdc37-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:05:39.415005 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.415007    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f0932d7-d1f4-4897-92a6-6297122fdc37-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:05:39.415005 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.415016    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgtf\" (UniqueName: \"kubernetes.io/projected/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-api-access-dcgtf\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:05:39.415265 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:39.415024    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f0932d7-d1f4-4897-92a6-6297122fdc37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:05:40.207881 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.207853    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"
Sep 17 19:05:40.207881 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.207861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb" event={"ID":"7f0932d7-d1f4-4897-92a6-6297122fdc37","Type":"ContainerDied","Data":"5a0b50ef6d7a2189335c380d837b329a0d3bc325d6af7303b294d00ce9c8c10a"}
Sep 17 19:05:40.208345 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.207922    2565 scope.go:117] "RemoveContainer" containerID="65f30b14ed3113418edabdab9e3bef3fc2e8199e9d5193d3e62a7b2e72514175"
Sep 17 19:05:40.216317 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.216302    2565 scope.go:117] "RemoveContainer" containerID="91d35e0993fa870845729a25dfcf8fbc15a0b8542faef96bc7a727b88349ea2d"
Sep 17 19:05:40.223428 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.223412    2565 scope.go:117] "RemoveContainer" containerID="38392937282f3a9a42c8405052ac42c20783debe3e7b4588702089d71cf3fc3b"
Sep 17 19:05:40.233986 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.233962    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"]
Sep 17 19:05:40.247634 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:40.247611    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-42kcb"]
Sep 17 19:05:41.062161 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:05:41.062127    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" path="/var/lib/kubelet/pods/7f0932d7-d1f4-4897-92a6-6297122fdc37/volumes"
Sep 17 19:07:04.853559 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:04.853476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"]
Sep 17 19:07:04.854084 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:04.853705    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" containerID="cri-o://9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2" gracePeriod=30
Sep 17 19:07:04.854084 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:04.853756    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kube-rbac-proxy" containerID="cri-o://974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97" gracePeriod=30
Sep 17 19:07:05.522734 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:05.522703    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerID="974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97" exitCode=2
Sep 17 19:07:05.522899 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:05.522778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerDied","Data":"974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97"}
Sep 17 19:07:07.870699 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:07.870663    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 19:07:07.876135 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:07.876111    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 19:07:09.303168 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.303140    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:07:09.483176 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-proxy-tls\") pod \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") "
Sep 17 19:07:09.483333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l6vs\" (UniqueName: \"kubernetes.io/projected/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-api-access-7l6vs\") pod \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") "
Sep 17 19:07:09.483333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483219    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-cabundle-cert\") pod \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") "
Sep 17 19:07:09.483333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kserve-provision-location\") pod \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") "
Sep 17 19:07:09.483333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483303    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-rbac-proxy-sar-config\") pod \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\" (UID: \"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d\") "
Sep 17 19:07:09.483668 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483639    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" (UID: "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:09.483834 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483744    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" (UID: "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:09.483834 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.483755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" (UID: "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:09.485847 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.485827    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" (UID: "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:09.485955 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.485829    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-api-access-7l6vs" (OuterVolumeSpecName: "kube-api-access-7l6vs") pod "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" (UID: "ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d"). InnerVolumeSpecName "kube-api-access-7l6vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:09.538045 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.538013    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerID="9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2" exitCode=0
Sep 17 19:07:09.538180 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.538067    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"
Sep 17 19:07:09.538180 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.538074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerDied","Data":"9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2"}
Sep 17 19:07:09.538252 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.538190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf" event={"ID":"ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d","Type":"ContainerDied","Data":"f68a5857f4461bbf755b3d35982eb62f152fd21f4ceaa90f2449e0345f72a3cb"}
Sep 17 19:07:09.538252 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.538218    2565 scope.go:117] "RemoveContainer" containerID="974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97"
Sep 17 19:07:09.546608 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.546590    2565 scope.go:117] "RemoveContainer" containerID="9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2"
Sep 17 19:07:09.553428 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.553409    2565 scope.go:117] "RemoveContainer" containerID="5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2"
Sep 17 19:07:09.560215 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.560199    2565 scope.go:117] "RemoveContainer" containerID="974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97"
Sep 17 19:07:09.560487 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:07:09.560469    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97\": container with ID starting with 974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97 not found: ID does not exist" containerID="974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97"
Sep 17 19:07:09.560546 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.560496    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97"} err="failed to get container status \"974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97\": rpc error: code = NotFound desc = could not find container \"974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97\": container with ID starting with 974da2e901a3a9ae4d45eec36ea106d423fc493e80199f55fb4be5c67cb08b97 not found: ID does not exist"
Sep 17 19:07:09.560546 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.560514    2565 scope.go:117] "RemoveContainer" containerID="9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2"
Sep 17 19:07:09.560767 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:07:09.560751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2\": container with ID starting with 9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2 not found: ID does not exist" containerID="9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2"
Sep 17 19:07:09.560817 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.560770    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2"} err="failed to get container status \"9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2\": rpc error: code = NotFound desc = could not find container \"9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2\": container with ID starting with 9f5b4b773d626e4b2a94750df9c90c84cf7e9185b827abc4e09baea8f10887e2 not found: ID does not exist"
Sep 17 19:07:09.560817 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.560782    2565 scope.go:117] "RemoveContainer" containerID="5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2"
Sep 17 19:07:09.561009 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:07:09.560994    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2\": container with ID starting with 5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2 not found: ID does not exist" containerID="5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2"
Sep 17 19:07:09.561050 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.561011    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2"} err="failed to get container status \"5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2\": rpc error: code = NotFound desc = could not find container \"5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2\": container with ID starting with 5c7f08989a8418541ad484713fe7753388b801aae36f36b3985b40f1eb5e34a2 not found: ID does not exist"
Sep 17 19:07:09.584384 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.584362    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.584384 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.584383    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6vs\" (UniqueName: \"kubernetes.io/projected/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-api-access-7l6vs\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.584510 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.584392    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.584510 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.584402    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.584510 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:09.584410    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:07:10.531172 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:10.531139    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"]
Sep 17 19:07:11.348396 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:11.348359    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-t6xnf"]
Sep 17 19:07:13.061480 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:13.061426    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" path="/var/lib/kubelet/pods/ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d/volumes"
Sep 17 19:07:20.444040 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444007    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"]
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444269    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="storage-initializer"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444280    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="storage-initializer"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444290    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444298    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444304    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kube-rbac-proxy"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444310    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kube-rbac-proxy"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444320    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kube-rbac-proxy"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444325    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kube-rbac-proxy"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444335    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="storage-initializer"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444340    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="storage-initializer"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444351    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444356    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444397    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kserve-container"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444404    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kserve-container"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444409    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0932d7-d1f4-4897-92a6-6297122fdc37" containerName="kube-rbac-proxy"
Sep 17 19:07:20.444415 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.444414    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6e00ed-ab03-477c-b9ae-7cc4043c4f8d" containerName="kube-rbac-proxy"
Sep 17 19:07:20.448511 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.448493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.450901 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.450880    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 19:07:20.451127 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.451109    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 19:07:20.451229 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.451206    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 19:07:20.451888 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.451872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:07:20.451970 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.451922    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:07:20.458057 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.458035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"]
Sep 17 19:07:20.564094 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.564064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.564256 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.564125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzwx\" (UniqueName: \"kubernetes.io/projected/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-api-access-pvzwx\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.564256 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.564162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f6aee4f-36df-4c0c-b96c-53c1cf710683-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.665162 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.665121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzwx\" (UniqueName: \"kubernetes.io/projected/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-api-access-pvzwx\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.665317 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.665171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f6aee4f-36df-4c0c-b96c-53c1cf710683-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.665317 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.665212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.665888 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.665864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.667604 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.667578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f6aee4f-36df-4c0c-b96c-53c1cf710683-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.674618 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.674600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzwx\" (UniqueName: \"kubernetes.io/projected/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-api-access-pvzwx\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-jx5z7\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:20.759092 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:20.759013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:21.028972 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:21.028949    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"]
Sep 17 19:07:21.031452 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:07:21.031415    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6aee4f_36df_4c0c_b96c_53c1cf710683.slice/crio-73739fe69d745db23fb93c5ed41c28cec75eb5aa49d3fcc3ebaf3c2028e4bfb3 WatchSource:0}: Error finding container 73739fe69d745db23fb93c5ed41c28cec75eb5aa49d3fcc3ebaf3c2028e4bfb3: Status 404 returned error can't find the container with id 73739fe69d745db23fb93c5ed41c28cec75eb5aa49d3fcc3ebaf3c2028e4bfb3
Sep 17 19:07:21.033355 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:21.033340    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:07:21.583593 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:21.583561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" event={"ID":"7f6aee4f-36df-4c0c-b96c-53c1cf710683","Type":"ContainerStarted","Data":"73739fe69d745db23fb93c5ed41c28cec75eb5aa49d3fcc3ebaf3c2028e4bfb3"}
Sep 17 19:07:22.588935 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:22.588899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" event={"ID":"7f6aee4f-36df-4c0c-b96c-53c1cf710683","Type":"ContainerStarted","Data":"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"}
Sep 17 19:07:22.589301 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:22.588960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" event={"ID":"7f6aee4f-36df-4c0c-b96c-53c1cf710683","Type":"ContainerStarted","Data":"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"}
Sep 17 19:07:22.589301 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:22.589004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:22.618555 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:22.618509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" podStartSLOduration=1.482700172 podStartE2EDuration="2.618491504s" totalImagesPullingTime="1.135791332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:22.618263513 +0000 UTC m=+1731.904576030" watchObservedRunningTime="2026-09-17 19:07:22.618491504 +0000 UTC m=+1731.904804022"
Sep 17 19:07:23.592641 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:23.592602    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:23.594430 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:23.594407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:30.603758 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:30.603733    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:07:40.411936 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.411907    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"]
Sep 17 19:07:40.416730 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.416709    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.419162 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.419141    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:07:40.419371 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.419356    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 19:07:40.419468 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.419382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 19:07:40.420095 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.420081    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:07:40.433155 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.433132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"]
Sep 17 19:07:40.501328 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.501298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81124c05-ec9f-4754-8a3b-c85444d843c5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.501489 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.501337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ckm\" (UniqueName: \"kubernetes.io/projected/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-api-access-k4ckm\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.501489 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.501383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81124c05-ec9f-4754-8a3b-c85444d843c5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.501489 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.501405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.501489 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.501426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.602617 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.602591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ckm\" (UniqueName: \"kubernetes.io/projected/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-api-access-k4ckm\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.602764 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.602640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81124c05-ec9f-4754-8a3b-c85444d843c5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.602764 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.602663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.602764 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.602731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.602937 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.602797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81124c05-ec9f-4754-8a3b-c85444d843c5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.603171 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.603143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81124c05-ec9f-4754-8a3b-c85444d843c5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.603453 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.603415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.603560 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.603415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.605261 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.605239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81124c05-ec9f-4754-8a3b-c85444d843c5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.612353 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.612327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ckm\" (UniqueName: \"kubernetes.io/projected/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-api-access-k4ckm\") pod \"isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:40.726797 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:40.726771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:41.004703 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:41.004677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"]
Sep 17 19:07:41.006975 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:07:41.006951    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81124c05_ec9f_4754_8a3b_c85444d843c5.slice/crio-427c51e42ca1d7247911dfc1ca9e778198e64ccc5f7b70897001f7ec617cd903 WatchSource:0}: Error finding container 427c51e42ca1d7247911dfc1ca9e778198e64ccc5f7b70897001f7ec617cd903: Status 404 returned error can't find the container with id 427c51e42ca1d7247911dfc1ca9e778198e64ccc5f7b70897001f7ec617cd903
Sep 17 19:07:41.657966 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:41.657925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerStarted","Data":"f464de9b1c11f0da2f71560e6af21c17086f16dbed84b86f9b9be52d793a08c3"}
Sep 17 19:07:41.658351 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:41.658009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerStarted","Data":"427c51e42ca1d7247911dfc1ca9e778198e64ccc5f7b70897001f7ec617cd903"}
Sep 17 19:07:45.676520 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:45.676486    2565 generic.go:356] "Generic (PLEG): container finished" podID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerID="f464de9b1c11f0da2f71560e6af21c17086f16dbed84b86f9b9be52d793a08c3" exitCode=0
Sep 17 19:07:45.676901 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:45.676564    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerDied","Data":"f464de9b1c11f0da2f71560e6af21c17086f16dbed84b86f9b9be52d793a08c3"}
Sep 17 19:07:46.682533 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.682499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerStarted","Data":"348b7acac716c81db8cba1f61dacd3726ab7dd598895a9ba5493d846a1692c17"}
Sep 17 19:07:46.682924 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.682573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerStarted","Data":"3e604d0e9d471c91de0e920bd5e6b08ccd9796eaf5edec678ccc74820ba74f65"}
Sep 17 19:07:46.682924 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.682603    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerStarted","Data":"a3c0d000e1d230c63be0b6ec9bcf29195c323f6cc6ba82bb72d7412b113075a0"}
Sep 17 19:07:46.682924 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.682907    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:46.683098 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.683062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:46.684523 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.684485    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:07:46.706584 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:46.706534    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podStartSLOduration=2.8883772629999998 podStartE2EDuration="6.706516746s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.818139483s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:46.705009204 +0000 UTC m=+1755.991321722" watchObservedRunningTime="2026-09-17 19:07:46.706516746 +0000 UTC m=+1755.992829263"
Sep 17 19:07:47.686766 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:47.686729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:47.687237 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:47.686865    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:07:47.688080 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:47.688049    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:07:48.691109 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:48.691070    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:07:48.691496 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:48.691404    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:07:53.695784 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:53.695755    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:07:53.696239 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:53.696212    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:07:53.696592 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:07:53.696556    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:08:03.696493 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:03.696428    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:08:03.696985 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:03.696963    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:08:13.696246 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:13.696206    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:08:13.696801 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:13.696766    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:08:23.696384 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:23.696294    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:08:23.696854 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:23.696828    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:08:33.696629 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:33.696588    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:08:33.697087 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:33.697061    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:08:43.696354 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:43.696313    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:08:43.696817 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:43.696727    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:08:53.697119 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:53.697076    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:08:53.697616 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:08:53.697558    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:09:03.061846 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:03.061813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:03.062218 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:03.061902    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:15.403096 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.403065    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-jx5z7_7f6aee4f-36df-4c0c-b96c-53c1cf710683/kserve-container/0.log"
Sep 17 19:09:15.679285 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.679206    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"]
Sep 17 19:09:15.679473 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.679451    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" containerID="cri-o://a3c0d000e1d230c63be0b6ec9bcf29195c323f6cc6ba82bb72d7412b113075a0" gracePeriod=30
Sep 17 19:09:15.679535 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.679487    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" containerID="cri-o://348b7acac716c81db8cba1f61dacd3726ab7dd598895a9ba5493d846a1692c17" gracePeriod=30
Sep 17 19:09:15.679592 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.679524    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" containerID="cri-o://3e604d0e9d471c91de0e920bd5e6b08ccd9796eaf5edec678ccc74820ba74f65" gracePeriod=30
Sep 17 19:09:15.737414 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.737376    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"]
Sep 17 19:09:15.737676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.737653    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kserve-container" containerID="cri-o://1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602" gracePeriod=30
Sep 17 19:09:15.737782 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.737672    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kube-rbac-proxy" containerID="cri-o://5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab" gracePeriod=30
Sep 17 19:09:15.980879 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.980857    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:09:15.988223 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.988203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-rbac-proxy-sar-config\") pod \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") "
Sep 17 19:09:15.988305 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.988243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvzwx\" (UniqueName: \"kubernetes.io/projected/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-api-access-pvzwx\") pod \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") "
Sep 17 19:09:15.988305 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.988270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f6aee4f-36df-4c0c-b96c-53c1cf710683-proxy-tls\") pod \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\" (UID: \"7f6aee4f-36df-4c0c-b96c-53c1cf710683\") "
Sep 17 19:09:15.988530 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.988508    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f6aee4f-36df-4c0c-b96c-53c1cf710683" (UID: "7f6aee4f-36df-4c0c-b96c-53c1cf710683"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:15.990190 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.990171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-api-access-pvzwx" (OuterVolumeSpecName: "kube-api-access-pvzwx") pod "7f6aee4f-36df-4c0c-b96c-53c1cf710683" (UID: "7f6aee4f-36df-4c0c-b96c-53c1cf710683"). InnerVolumeSpecName "kube-api-access-pvzwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:15.990284 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:15.990248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6aee4f-36df-4c0c-b96c-53c1cf710683-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f6aee4f-36df-4c0c-b96c-53c1cf710683" (UID: "7f6aee4f-36df-4c0c-b96c-53c1cf710683"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.018873 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.018846    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerID="5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab" exitCode=2
Sep 17 19:09:16.018873 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.018865    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerID="1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602" exitCode=2
Sep 17 19:09:16.019041 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.018883    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"
Sep 17 19:09:16.019041 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.018926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" event={"ID":"7f6aee4f-36df-4c0c-b96c-53c1cf710683","Type":"ContainerDied","Data":"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"}
Sep 17 19:09:16.019041 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.018995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" event={"ID":"7f6aee4f-36df-4c0c-b96c-53c1cf710683","Type":"ContainerDied","Data":"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"}
Sep 17 19:09:16.019041 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.019030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7" event={"ID":"7f6aee4f-36df-4c0c-b96c-53c1cf710683","Type":"ContainerDied","Data":"73739fe69d745db23fb93c5ed41c28cec75eb5aa49d3fcc3ebaf3c2028e4bfb3"}
Sep 17 19:09:16.019248 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.019067    2565 scope.go:117] "RemoveContainer" containerID="5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"
Sep 17 19:09:16.022107 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.022084    2565 generic.go:356] "Generic (PLEG): container finished" podID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerID="3e604d0e9d471c91de0e920bd5e6b08ccd9796eaf5edec678ccc74820ba74f65" exitCode=2
Sep 17 19:09:16.022193 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.022117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerDied","Data":"3e604d0e9d471c91de0e920bd5e6b08ccd9796eaf5edec678ccc74820ba74f65"}
Sep 17 19:09:16.027663 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.027581    2565 scope.go:117] "RemoveContainer" containerID="1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"
Sep 17 19:09:16.034534 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.034517    2565 scope.go:117] "RemoveContainer" containerID="5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"
Sep 17 19:09:16.034776 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:09:16.034756    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab\": container with ID starting with 5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab not found: ID does not exist" containerID="5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"
Sep 17 19:09:16.034841 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.034787    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"} err="failed to get container status \"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab\": rpc error: code = NotFound desc = could not find container \"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab\": container with ID starting with 5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab not found: ID does not exist"
Sep 17 19:09:16.034841 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.034809    2565 scope.go:117] "RemoveContainer" containerID="1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"
Sep 17 19:09:16.035068 ip-10-0-130-95 kubenswrapper[2565]: E0917 19:09:16.035053    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602\": container with ID starting with 1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602 not found: ID does not exist" containerID="1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"
Sep 17 19:09:16.035109 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.035076    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"} err="failed to get container status \"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602\": rpc error: code = NotFound desc = could not find container \"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602\": container with ID starting with 1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602 not found: ID does not exist"
Sep 17 19:09:16.035109 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.035091    2565 scope.go:117] "RemoveContainer" containerID="5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"
Sep 17 19:09:16.035287 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.035269    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab"} err="failed to get container status \"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab\": rpc error: code = NotFound desc = could not find container \"5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab\": container with ID starting with 5fce16d0a764c1759f5202477050184ecbc9939a0d0305079c07d9d3319523ab not found: ID does not exist"
Sep 17 19:09:16.035333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.035289    2565 scope.go:117] "RemoveContainer" containerID="1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"
Sep 17 19:09:16.035548 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.035529    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602"} err="failed to get container status \"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602\": rpc error: code = NotFound desc = could not find container \"1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602\": container with ID starting with 1769bee8a8b48da2accf611784d686d1644e9c8c405c631109b20ed193e23602 not found: ID does not exist"
Sep 17 19:09:16.042487 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.042466    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"]
Sep 17 19:09:16.046387 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.046368    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-jx5z7"]
Sep 17 19:09:16.089323 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.089306    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:16.089323 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.089324    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvzwx\" (UniqueName: \"kubernetes.io/projected/7f6aee4f-36df-4c0c-b96c-53c1cf710683-kube-api-access-pvzwx\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:16.089478 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:16.089334    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f6aee4f-36df-4c0c-b96c-53c1cf710683-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.062349 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:17.062276    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" path="/var/lib/kubelet/pods/7f6aee4f-36df-4c0c-b96c-53c1cf710683/volumes"
Sep 17 19:09:18.692200 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:18.692161    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 19:09:20.040316 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:20.040289    2565 generic.go:356] "Generic (PLEG): container finished" podID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerID="a3c0d000e1d230c63be0b6ec9bcf29195c323f6cc6ba82bb72d7412b113075a0" exitCode=0
Sep 17 19:09:20.040630 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:20.040362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerDied","Data":"a3c0d000e1d230c63be0b6ec9bcf29195c323f6cc6ba82bb72d7412b113075a0"}
Sep 17 19:09:23.058956 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:23.058918    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:09:23.059416 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:23.059365    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:09:23.692026 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:23.691982    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 19:09:28.691681 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:28.691640    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 19:09:28.693960 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:28.691787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:31.918488 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:31.918429    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:31.942750 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:31.942724    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:31.952612 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:31.952590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:31.962766 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:31.962745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:32.816152 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:32.816124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:32.829727 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:32.829701    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:32.839421 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:32.839398    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:32.849154 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:32.849136    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:33.058976 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.058939    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:09:33.059452 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.059396    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:09:33.644310 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.644199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:33.656743 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.656717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:33.667388 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.667369    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:33.680745 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.680720    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:33.691321 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:33.691292    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 19:09:34.451987 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:34.451959    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:34.463588 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:34.463567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:34.473427 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:34.473409    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:34.488189 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:34.488169    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:35.274186 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:35.274159    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:35.285838 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:35.285819    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:35.296800 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:35.296782    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:35.307593 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:35.307578    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:36.102676 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.102643    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:36.114422 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.114402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:36.124264 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.124245    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:36.134514 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.134499    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:36.925992 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.925966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:36.938940 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.938917    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:36.950069 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.950049    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:36.961806 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:36.961786    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:37.786002 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:37.785965    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:37.798324 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:37.798295    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:37.809801 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:37.809780    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:37.820986 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:37.820966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:38.659662 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:38.659630    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:38.671731 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:38.671711    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:38.682219 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:38.682194    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:38.691725 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:38.691696    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 19:09:38.692448 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:38.692406    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:39.591765 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:39.591735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:39.603547 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:39.603520    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:39.613809 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:39.613785    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:39.624060 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:39.624038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:40.423905 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:40.423870    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:40.436843 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:40.436821    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:40.449564 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:40.449534    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:40.460714 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:40.460677    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:41.302396 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:41.302371    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:41.314398 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:41.314377    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:41.325423 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:41.325404    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:41.335969 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:41.335952    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:42.121036 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.121006    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:42.131904 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.131881    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:42.142230 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.142214    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:42.152472 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.152428    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:42.975300 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.975271    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:42.987358 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.987338    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:42.998524 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:42.998503    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:43.009938 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.009918    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:43.058956 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.058932    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:09:43.059333 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.059304    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:09:43.062014 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.061998    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:43.062092 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.062072    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:43.691894 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.691857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 19:09:43.900179 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.900152    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kserve-container/0.log"
Sep 17 19:09:43.911864 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.911845    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/kube-rbac-proxy/0.log"
Sep 17 19:09:43.922656 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.922634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/agent/0.log"
Sep 17 19:09:43.933202 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:43.933184    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr_81124c05-ec9f-4754-8a3b-c85444d843c5/storage-initializer/0.log"
Sep 17 19:09:46.135111 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.135029    2565 generic.go:356] "Generic (PLEG): container finished" podID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerID="348b7acac716c81db8cba1f61dacd3726ab7dd598895a9ba5493d846a1692c17" exitCode=0
Sep 17 19:09:46.135111 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.135094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerDied","Data":"348b7acac716c81db8cba1f61dacd3726ab7dd598895a9ba5493d846a1692c17"}
Sep 17 19:09:46.319282 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.319257    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:46.395325 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81124c05-ec9f-4754-8a3b-c85444d843c5-kserve-provision-location\") pod \"81124c05-ec9f-4754-8a3b-c85444d843c5\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") "
Sep 17 19:09:46.395325 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-rbac-proxy-sar-config\") pod \"81124c05-ec9f-4754-8a3b-c85444d843c5\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") "
Sep 17 19:09:46.395559 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81124c05-ec9f-4754-8a3b-c85444d843c5-proxy-tls\") pod \"81124c05-ec9f-4754-8a3b-c85444d843c5\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") "
Sep 17 19:09:46.395559 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395359    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-cabundle-cert\") pod \"81124c05-ec9f-4754-8a3b-c85444d843c5\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") "
Sep 17 19:09:46.395559 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395382    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4ckm\" (UniqueName: \"kubernetes.io/projected/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-api-access-k4ckm\") pod \"81124c05-ec9f-4754-8a3b-c85444d843c5\" (UID: \"81124c05-ec9f-4754-8a3b-c85444d843c5\") "
Sep 17 19:09:46.395559 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81124c05-ec9f-4754-8a3b-c85444d843c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81124c05-ec9f-4754-8a3b-c85444d843c5" (UID: "81124c05-ec9f-4754-8a3b-c85444d843c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:46.395775 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395562    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81124c05-ec9f-4754-8a3b-c85444d843c5-kserve-provision-location\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:46.395775 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395654    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81124c05-ec9f-4754-8a3b-c85444d843c5" (UID: "81124c05-ec9f-4754-8a3b-c85444d843c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:46.395775 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.395695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81124c05-ec9f-4754-8a3b-c85444d843c5" (UID: "81124c05-ec9f-4754-8a3b-c85444d843c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:46.397505 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.397413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81124c05-ec9f-4754-8a3b-c85444d843c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81124c05-ec9f-4754-8a3b-c85444d843c5" (UID: "81124c05-ec9f-4754-8a3b-c85444d843c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:46.397505 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.397421    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-api-access-k4ckm" (OuterVolumeSpecName: "kube-api-access-k4ckm") pod "81124c05-ec9f-4754-8a3b-c85444d843c5" (UID: "81124c05-ec9f-4754-8a3b-c85444d843c5"). InnerVolumeSpecName "kube-api-access-k4ckm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:46.496136 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.496097    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:46.496136 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.496133    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81124c05-ec9f-4754-8a3b-c85444d843c5-proxy-tls\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:46.496281 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.496149    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81124c05-ec9f-4754-8a3b-c85444d843c5-cabundle-cert\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:46.496281 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:46.496161    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4ckm\" (UniqueName: \"kubernetes.io/projected/81124c05-ec9f-4754-8a3b-c85444d843c5-kube-api-access-k4ckm\") on node \"ip-10-0-130-95.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.140479 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.140431    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"
Sep 17 19:09:47.140816 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.140431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr" event={"ID":"81124c05-ec9f-4754-8a3b-c85444d843c5","Type":"ContainerDied","Data":"427c51e42ca1d7247911dfc1ca9e778198e64ccc5f7b70897001f7ec617cd903"}
Sep 17 19:09:47.140816 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.140595    2565 scope.go:117] "RemoveContainer" containerID="348b7acac716c81db8cba1f61dacd3726ab7dd598895a9ba5493d846a1692c17"
Sep 17 19:09:47.148270 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.148253    2565 scope.go:117] "RemoveContainer" containerID="3e604d0e9d471c91de0e920bd5e6b08ccd9796eaf5edec678ccc74820ba74f65"
Sep 17 19:09:47.155293 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.155275    2565 scope.go:117] "RemoveContainer" containerID="a3c0d000e1d230c63be0b6ec9bcf29195c323f6cc6ba82bb72d7412b113075a0"
Sep 17 19:09:47.161979 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.161962    2565 scope.go:117] "RemoveContainer" containerID="f464de9b1c11f0da2f71560e6af21c17086f16dbed84b86f9b9be52d793a08c3"
Sep 17 19:09:47.184845 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.184824    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"]
Sep 17 19:09:47.191752 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:47.191733    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c98dccfb-g4bvr"]
Sep 17 19:09:48.075945 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.075914    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-59s4v/must-gather-wjmcv"]
Sep 17 19:09:48.076619 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076582    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container"
Sep 17 19:09:48.076619 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076610    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076638    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076647    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076676    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="storage-initializer"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076684    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="storage-initializer"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076701    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076709    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076770    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kserve-container"
Sep 17 19:09:48.076795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076786    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kserve-container"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076800    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kube-rbac-proxy"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076809    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kube-rbac-proxy"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076979    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kube-rbac-proxy"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.076991    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kserve-container"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.077009    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6aee4f-36df-4c0c-b96c-53c1cf710683" containerName="kserve-container"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.077021    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="agent"
Sep 17 19:09:48.077072 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.077031    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" containerName="kube-rbac-proxy"
Sep 17 19:09:48.082624 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.082601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.085702 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.085682    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59s4v\"/\"openshift-service-ca.crt\""
Sep 17 19:09:48.085814 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.085682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-59s4v\"/\"default-dockercfg-v4fdh\""
Sep 17 19:09:48.085814 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.085722    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59s4v\"/\"kube-root-ca.crt\""
Sep 17 19:09:48.091051 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.091032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59s4v/must-gather-wjmcv"]
Sep 17 19:09:48.106732 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.106704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f2d781f-778a-42e7-99c6-53de83844de7-must-gather-output\") pod \"must-gather-wjmcv\" (UID: \"2f2d781f-778a-42e7-99c6-53de83844de7\") " pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.106811 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.106760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6q2\" (UniqueName: \"kubernetes.io/projected/2f2d781f-778a-42e7-99c6-53de83844de7-kube-api-access-nf6q2\") pod \"must-gather-wjmcv\" (UID: \"2f2d781f-778a-42e7-99c6-53de83844de7\") " pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.207736 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.207708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f2d781f-778a-42e7-99c6-53de83844de7-must-gather-output\") pod \"must-gather-wjmcv\" (UID: \"2f2d781f-778a-42e7-99c6-53de83844de7\") " pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.208011 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.207772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6q2\" (UniqueName: \"kubernetes.io/projected/2f2d781f-778a-42e7-99c6-53de83844de7-kube-api-access-nf6q2\") pod \"must-gather-wjmcv\" (UID: \"2f2d781f-778a-42e7-99c6-53de83844de7\") " pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.208081 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.208063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f2d781f-778a-42e7-99c6-53de83844de7-must-gather-output\") pod \"must-gather-wjmcv\" (UID: \"2f2d781f-778a-42e7-99c6-53de83844de7\") " pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.218180 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.218158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6q2\" (UniqueName: \"kubernetes.io/projected/2f2d781f-778a-42e7-99c6-53de83844de7-kube-api-access-nf6q2\") pod \"must-gather-wjmcv\" (UID: \"2f2d781f-778a-42e7-99c6-53de83844de7\") " pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.392698 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.392627    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59s4v/must-gather-wjmcv"
Sep 17 19:09:48.659666 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:48.659587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59s4v/must-gather-wjmcv"]
Sep 17 19:09:48.662194 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:09:48.662166    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f2d781f_778a_42e7_99c6_53de83844de7.slice/crio-43f278df56286c815ed5e429f5f226361347b5c13c6126d2a7f3b51e9c0461b8 WatchSource:0}: Error finding container 43f278df56286c815ed5e429f5f226361347b5c13c6126d2a7f3b51e9c0461b8: Status 404 returned error can't find the container with id 43f278df56286c815ed5e429f5f226361347b5c13c6126d2a7f3b51e9c0461b8
Sep 17 19:09:49.061636 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:49.061603    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81124c05-ec9f-4754-8a3b-c85444d843c5" path="/var/lib/kubelet/pods/81124c05-ec9f-4754-8a3b-c85444d843c5/volumes"
Sep 17 19:09:49.149266 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:49.149232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/must-gather-wjmcv" event={"ID":"2f2d781f-778a-42e7-99c6-53de83844de7","Type":"ContainerStarted","Data":"43f278df56286c815ed5e429f5f226361347b5c13c6126d2a7f3b51e9c0461b8"}
Sep 17 19:09:50.154868 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:50.154832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/must-gather-wjmcv" event={"ID":"2f2d781f-778a-42e7-99c6-53de83844de7","Type":"ContainerStarted","Data":"5e96f75d73f9f482b7af6e81391f7d329ab882fb4a1b869fd4f2bd91acdf0316"}
Sep 17 19:09:50.155236 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:50.154876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/must-gather-wjmcv" event={"ID":"2f2d781f-778a-42e7-99c6-53de83844de7","Type":"ContainerStarted","Data":"63c77d828e8e7e8233bdd5b5a7b96ac68ef6ac4aa7f57078b42eb9b0214c1be6"}
Sep 17 19:09:50.172924 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:50.172866    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-59s4v/must-gather-wjmcv" podStartSLOduration=1.3913581050000001 podStartE2EDuration="2.172846483s" totalImagesPullingTime="781.488378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:50.171487792 +0000 UTC m=+1879.457800306" watchObservedRunningTime="2026-09-17 19:09:50.172846483 +0000 UTC m=+1879.459159001"
Sep 17 19:09:50.875567 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:50.875515    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gvpvp_433bbf08-f013-4ad0-8fd0-f592eba5ac4c/global-pull-secret-syncer/0.log"
Sep 17 19:09:51.031828 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:51.031788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vjx5b_41f96877-8462-4f9c-a21e-1cefab8b45cd/konnectivity-agent/0.log"
Sep 17 19:09:51.054049 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:51.054021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-95.ec2.internal_8f3ea4320296e328c2dbf6e5b180d445/haproxy/0.log"
Sep 17 19:09:55.247845 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:55.247817    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-ddvnb_9d4f84b2-13fa-465c-bca2-aa7f8027c5f5/cluster-monitoring-operator/0.log"
Sep 17 19:09:55.446148 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:55.446108    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtbbg_d1d5c0d8-4c6a-4cb6-85f5-342477ef2018/node-exporter/0.log"
Sep 17 19:09:55.472608 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:55.472562    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtbbg_d1d5c0d8-4c6a-4cb6-85f5-342477ef2018/kube-rbac-proxy/0.log"
Sep 17 19:09:55.498481 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:55.498337    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtbbg_d1d5c0d8-4c6a-4cb6-85f5-342477ef2018/init-textfile/0.log"
Sep 17 19:09:58.984026 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:58.983991    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"]
Sep 17 19:09:58.990831 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:58.990799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.004795 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.004765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"]
Sep 17 19:09:59.078230 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.078200    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-rg9g7_61c70057-1bab-4c67-9d06-60c5d910329a/volume-data-source-validator/0.log"
Sep 17 19:09:59.109181 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.109148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-proc\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.109356 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.109193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-podres\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.109356 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.109243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-lib-modules\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.109356 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.109268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572cj\" (UniqueName: \"kubernetes.io/projected/5ee35a4c-da3d-440a-8c87-b4903d52263b-kube-api-access-572cj\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.109356 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.109289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-sys\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.209778 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-podres\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.209961 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-lib-modules\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.209961 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572cj\" (UniqueName: \"kubernetes.io/projected/5ee35a4c-da3d-440a-8c87-b4903d52263b-kube-api-access-572cj\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.209961 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-sys\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.209961 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-proc\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.209961 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-podres\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.210222 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-proc\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.210222 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.209990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-lib-modules\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.210222 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.210011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee35a4c-da3d-440a-8c87-b4903d52263b-sys\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.219047 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.219021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572cj\" (UniqueName: \"kubernetes.io/projected/5ee35a4c-da3d-440a-8c87-b4903d52263b-kube-api-access-572cj\") pod \"perf-node-gather-daemonset-9xpzt\" (UID: \"5ee35a4c-da3d-440a-8c87-b4903d52263b\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.303955 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.303878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:09:59.587533 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:09:59.587425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"]
Sep 17 19:09:59.591333 ip-10-0-130-95 kubenswrapper[2565]: W0917 19:09:59.591293    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ee35a4c_da3d_440a_8c87_b4903d52263b.slice/crio-33135621f64e474709cb11448fe6d42701229cded7685bf8cd440d8bbebff6d7 WatchSource:0}: Error finding container 33135621f64e474709cb11448fe6d42701229cded7685bf8cd440d8bbebff6d7: Status 404 returned error can't find the container with id 33135621f64e474709cb11448fe6d42701229cded7685bf8cd440d8bbebff6d7
Sep 17 19:10:00.000365 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.000333    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g9jcp_0bc865a8-e383-4521-b9ff-0c2a3c0bdbff/dns/0.log"
Sep 17 19:10:00.022414 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.022384    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g9jcp_0bc865a8-e383-4521-b9ff-0c2a3c0bdbff/kube-rbac-proxy/0.log"
Sep 17 19:10:00.166646 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.166619    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-99thn_be519205-5d21-460c-89ed-53124a87b5fe/dns-node-resolver/0.log"
Sep 17 19:10:00.205636 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.205603    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt" event={"ID":"5ee35a4c-da3d-440a-8c87-b4903d52263b","Type":"ContainerStarted","Data":"61c36b1d2bb2741e74c50a7d8a0afee7941075e223c0518c7a52dc18f6affe8f"}
Sep 17 19:10:00.205828 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.205689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt" event={"ID":"5ee35a4c-da3d-440a-8c87-b4903d52263b","Type":"ContainerStarted","Data":"33135621f64e474709cb11448fe6d42701229cded7685bf8cd440d8bbebff6d7"}
Sep 17 19:10:00.205828 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.205760    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:10:00.227353 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.227296    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt" podStartSLOduration=2.227277809 podStartE2EDuration="2.227277809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:00.224516563 +0000 UTC m=+1889.510829082" watchObservedRunningTime="2026-09-17 19:10:00.227277809 +0000 UTC m=+1889.513590327"
Sep 17 19:10:00.636769 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:00.636740    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8k74_16b01038-bf4c-4648-bf2c-b5a073762284/node-ca/0.log"
Sep 17 19:10:01.679523 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:01.679496    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpkk2_e4d4de15-fc5d-48c9-aa48-da635f088ef7/serve-healthcheck-canary/0.log"
Sep 17 19:10:02.183521 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:02.183473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2bgs_81b62802-e1c3-4cb0-9d7d-11d669fc6e32/kube-rbac-proxy/0.log"
Sep 17 19:10:02.202302 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:02.202278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2bgs_81b62802-e1c3-4cb0-9d7d-11d669fc6e32/exporter/0.log"
Sep 17 19:10:02.222166 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:02.222143    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2bgs_81b62802-e1c3-4cb0-9d7d-11d669fc6e32/extractor/0.log"
Sep 17 19:10:04.521136 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:04.521106    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-txl5t_76adb496-0ce6-4d02-9c8d-644b0608707a/server/0.log"
Sep 17 19:10:04.909981 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:04.909908    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-jqjjn_5556a267-b158-42be-bbcc-c5f28211c3fc/manager/0.log"
Sep 17 19:10:04.993161 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:04.993134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-s6t58_4a4b1d10-d492-40eb-a0f7-c8f49875f5c9/seaweedfs/0.log"
Sep 17 19:10:06.220574 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:06.220021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-9xpzt"
Sep 17 19:10:10.505206 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.505136    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:10:10.530169 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.530139    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/egress-router-binary-copy/0.log"
Sep 17 19:10:10.552996 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.552974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/cni-plugins/0.log"
Sep 17 19:10:10.575617 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.575582    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/bond-cni-plugin/0.log"
Sep 17 19:10:10.594377 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.594356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/routeoverride-cni/0.log"
Sep 17 19:10:10.614114 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.614089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/whereabouts-cni-bincopy/0.log"
Sep 17 19:10:10.632833 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:10.632813    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jnzg_7424f0ba-a16a-4291-936b-391bcb6fdb74/whereabouts-cni/0.log"
Sep 17 19:10:11.066122 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:11.066098    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qs64h_ccc26168-6aad-4eb1-a6df-db17738b8717/kube-multus/0.log"
Sep 17 19:10:11.294861 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:11.294835    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jhxpv_0255f4fa-4df9-439c-8cc2-5b921c8a64a1/network-metrics-daemon/0.log"
Sep 17 19:10:11.319116 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:11.319051    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jhxpv_0255f4fa-4df9-439c-8cc2-5b921c8a64a1/kube-rbac-proxy/0.log"
Sep 17 19:10:12.122809 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.122782    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/ovn-controller/0.log"
Sep 17 19:10:12.139758 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.139715    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/ovn-acl-logging/0.log"
Sep 17 19:10:12.164494 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.164463    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/kube-rbac-proxy-node/0.log"
Sep 17 19:10:12.184255 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.184237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:10:12.206269 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.206249    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/northd/0.log"
Sep 17 19:10:12.226793 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.226775    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/nbdb/0.log"
Sep 17 19:10:12.249152 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.249134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/sbdb/0.log"
Sep 17 19:10:12.289117 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:12.289100    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2lf85_4a439def-a26b-458c-952b-86efaf3a282c/ovnkube-controller/0.log"
Sep 17 19:10:13.607847 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:13.607815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-km4bg_dcf0ad40-19d2-44f9-a69d-911b05e056e9/check-endpoints/0.log"
Sep 17 19:10:13.651702 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:13.651679    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-pzcln_d63ac213-96df-4e51-9fd9-b76edfb86467/network-check-target-container/0.log"
Sep 17 19:10:14.511377 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:14.511344    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g4d9g_ad0ec7bd-4def-4f2c-8bc8-6a0e6371a42d/iptables-alerter/0.log"
Sep 17 19:10:15.189852 ip-10-0-130-95 kubenswrapper[2565]: I0917 19:10:15.189824    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-tzs2x_828d3595-96af-4fae-9952-f77f3e34a2be/tuned/0.log"