Sep 17 17:38:52.787711 ip-10-0-138-144 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:38:52.787847 ip-10-0-138-144 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:38:52.787945 ip-10-0-138-144 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:38:52.788616 ip-10-0-138-144 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:39:02.956163 ip-10-0-138-144 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:39:02.956184 ip-10-0-138-144 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 56cb277dc51c4befa9a2659f48949c56 --
Sep 17 17:41:05.250195 ip-10-0-138-144 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:41:08.782514 ip-10-0-138-144 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:08.782514 ip-10-0-138-144 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:41:08.782514 ip-10-0-138-144 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:08.782514 ip-10-0-138-144 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:41:08.782514 ip-10-0-138-144 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:08.783239 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.782588    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:41:08.786326 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786309    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786326    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786332    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786335    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786338    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786341    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786344    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786347    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786355    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786358    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786361    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786364    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786367    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786370    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:08.786365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786373    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786375    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786379    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786382    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786384    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786387    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786389    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786392    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786395    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786397    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786400    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786403    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786406    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786409    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786411    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786414    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786417    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786419    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786422    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786424    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:08.786737 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786427    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786431    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786434    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786436    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786439    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786442    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786446    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786448    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786451    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786454    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786457    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786459    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786462    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786475    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786479    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786482    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786484    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786487    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786490    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786493    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:08.787258 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786496    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786498    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786501    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786504    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786506    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786509    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786512    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786514    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786517    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786519    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786525    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786529    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786532    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786535    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786538    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786541    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786543    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786547    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786549    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786552    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:08.787775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786555    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786557    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786560    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786564    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786567    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786570    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786574    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786577    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786580    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786583    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786586    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786588    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786591    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786594    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786597    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786600    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786611    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786614    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786617    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:08.788309 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786620    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786622    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786625    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786628    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786631    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.786633    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787385    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787392    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787396    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787400    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787403    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787407    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787410    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787413    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787415    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787418    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787421    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787424    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787427    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787431    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:08.788806 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787434    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787437    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787440    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787443    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787446    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787448    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787451    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787454    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787456    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787459    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787462    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787464    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787467    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787470    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787474    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787477    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787480    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787483    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787485    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787488    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:08.789356 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787491    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787494    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787497    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787500    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787503    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787506    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787508    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787511    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787513    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787516    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787519    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787523    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787526    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787528    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787531    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787534    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787537    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787539    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787544    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787547    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:08.789901 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787551    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787553    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787556    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787559    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787562    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787564    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787568    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787571    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787573    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787576    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787580    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787584    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787587    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787590    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787593    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787596    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787598    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787602    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787605    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:08.790460 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787608    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787611    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787614    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787617    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787619    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787622    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787624    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787627    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787629    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787632    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787635    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787637    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787640    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787642    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787645    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787648    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787650    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787653    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787655    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:08.790988 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787658    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787661    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787663    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787666    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787668    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787671    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.787673    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788063    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788068    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788072    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788075    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788078    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788081    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788084    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788087    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788090    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788093    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788096    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788099    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:08.791478 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788102    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788104    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788107    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788110    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788112    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788117    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788121    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788124    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788127    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788130    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788133    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788136    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788139    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788142    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788145    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788148    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788151    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788153    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788156    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788159    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:08.792029 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788161    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788164    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788167    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788169    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788172    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788175    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788178    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788181    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788183    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788186    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788188    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788191    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788194    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788196    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788199    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788201    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788204    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788207    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788209    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788212    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:08.792590 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788214    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788217    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788220    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788223    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788225    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788228    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788230    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788233    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788236    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788238    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788241    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788244    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788246    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788252    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788254    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788257    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788259    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788264    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788267    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788270    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:08.793123 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788273    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788275    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788278    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788281    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788284    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788287    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788290    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788292    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788295    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788297    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788300    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788302    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788305    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788308    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788310    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788313    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788316    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788319    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788321    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:08.793681 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788324    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788328    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788332    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788334    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788337    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788340    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788343    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.788347    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788428    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788462    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788471    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788475    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788480    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788483    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788487    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788491    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788494    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788498    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788501    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788505    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788508    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:41:08.794174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788511    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788514    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788517    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788520    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788523    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788527    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788530    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788533    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788536    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788540    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788543    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788546    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788549    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788553    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788556    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788559    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788562    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788565    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788568    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788572    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788575    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788580    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788583    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788586    2585 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788590    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:41:08.794708 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788594    2585 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788597    2585 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788600    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788603    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788606    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788610    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788613    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788616    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788619    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788622    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788626    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788628    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788631    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788634    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788638    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788640    2585 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788644    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788647    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788651    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788654    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788657    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788660    2585 flags.go:64] FLAG: --help="false"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788663    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-138-144.ec2.internal"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788666    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:41:08.795494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788669    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788672    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788676    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788680    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788684    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788687    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788690    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788693    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788696    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788700    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788703    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788706    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788708    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788712    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788715    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788718    2585 flags.go:64] FLAG: --lock-file=""
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788721    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788724    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788727    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788733    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788736    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788739    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788742    2585 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:41:08.796167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788745    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788748    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788751    2585 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788754    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788758    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788761    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788765    2585 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788768    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788771    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788774    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788777    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788780    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788783    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788788    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788794    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788798    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788800    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788804    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788807    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788813    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788816    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788819    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788822    2585 flags.go:64] FLAG: --port="10250"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788825    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:41:08.796778 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788828    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09724c850547fde2f"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788831    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788836    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788839    2585 flags.go:64] FLAG: --register-node="true"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788842    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788845    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788848    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788851    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788854    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788858    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788861    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788864    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788867    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788870    2585 flags.go:64] FLAG: --runonce="false"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788872    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788876    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788879    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788898    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788901    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788904    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788907    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788910    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788915    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788918    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788921    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788924    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:41:08.797434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788927    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788930    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788933    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788939    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788942    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788945    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788948    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788951    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788955    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788958    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788961    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788964    2585 flags.go:64] FLAG: --v="2"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788968    2585 flags.go:64] FLAG: --version="false"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788972    2585 flags.go:64] FLAG: --vmodule=""
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788975    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.788979    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789095    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789100    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789103    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789107    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789110    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789113    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789116    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:08.798140 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789119    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789121    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789124    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789127    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789130    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789135    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789138    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789140    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789143    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789146    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789149    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789152    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789155    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789158    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789160    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789165    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789169    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789173    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789176    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:08.798741 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789178    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789181    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789184    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789187    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789189    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789192    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789195    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789198    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789200    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789203    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789206    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789208    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789211    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789214    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789217    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789219    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789222    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789225    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789230    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789233    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:08.799268 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789236    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789238    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789241    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789244    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789246    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789249    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789252    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789254    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789257    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789260    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789264    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789267    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789269    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789272    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789274    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789277    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789280    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789284    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789288    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789291    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:08.799809 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789293    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789296    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789299    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789301    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789304    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789307    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789309    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789312    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789316    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789320    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789324    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789327    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789330    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789333    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789336    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789339    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789341    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789344    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789347    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:08.800360 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789350    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789353    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789355    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789363    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789366    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789369    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789372    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789374    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789377    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789379    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789382    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789385    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789388    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.789391    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.789396    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:08.800907 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.796684    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.796698    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.796718    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.796723    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796778    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796783    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796787    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796790    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796793    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796796    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796799    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796802    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796805    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796807    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796810    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796813    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796815    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796818    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796821    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:08.801313 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796823    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796826    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796829    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796831    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796834    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796837    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796839    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796842    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796845    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796847    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796850    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796852    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796855    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796857    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796862    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796864    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796867    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796870    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796873    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796877    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:08.801811 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796881    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796908    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796913    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796924    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796926    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796929    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796932    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796935    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796937    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796940    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796943    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796946    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796948    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796951    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796953    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796956    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796959    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796961    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796964    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796966    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:08.802398 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796969    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796972    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796974    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796978    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796980    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796983    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796985    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796989    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796993    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796995    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.796998    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797001    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797004    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797007    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797009    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797012    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797015    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797017    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797021    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797026    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:08.802999 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797029    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797032    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797034    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797037    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797040    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797042    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797045    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797047    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797050    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797053    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797055    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797058    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797060    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797063    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797066    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797068    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797073    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797077    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797080    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:08.803778 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797083    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797086    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797089    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797092    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797095    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.797100    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797230    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797235    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797238    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797241    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797245    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797247    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797250    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797253    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797256    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:08.804659 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797259    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797262    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797264    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797267    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797270    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797272    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797275    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797278    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797280    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797283    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797285    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797288    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797291    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797293    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797296    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797298    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797302    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797305    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797307    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797310    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:08.805330 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797313    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797316    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797319    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797321    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797324    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797327    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797330    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797332    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797335    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797337    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797340    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797343    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797346    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797349    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797351    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797354    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797356    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797359    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797362    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797364    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:08.806133 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797367    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797369    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797372    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797375    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797377    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797380    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797382    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797386    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797390    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797394    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797397    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797400    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797403    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797406    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797409    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797412    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797415    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797418    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797421    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:08.807050 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797424    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797427    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797429    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797432    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797434    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797437    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797440    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797442    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797446    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797450    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797453    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797456    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797458    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797461    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797464    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797466    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797469    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797472    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797474    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797477    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:08.807767 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797480    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797483    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797486    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797488    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797491    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797494    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797497    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797500    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797502    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797505    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:08.797507    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.797512    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.797610    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.799691    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:41:08.808318 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.800453    2585 server.go:1019] "Starting client certificate rotation"
Sep 17 17:41:08.808714 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.800559    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:08.808714 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.801240    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:08.808714 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.804236    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:08.808714 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.807139    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:08.811508 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.811492    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:41:08.813554 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.813538    2585 log.go:25] "Validated CRI v1 image API"
Sep 17 17:41:08.815059 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.815043    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:41:08.819475 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.819456    2585 fs.go:135] Filesystem UUIDs: map[542cdf35-0c77-4aff-a902-48b9a332d4df:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7c78577b-0838-4603-9bdb-93c98c046627:/dev/nvme0n1p4]
Sep 17 17:41:08.819521 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.819476    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:41:08.824731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.824619    2585 manager.go:216] Machine: {Timestamp:2026-09-17 17:41:08.82332305 +0000 UTC m=+2.014660729 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101838 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c17f08d867bab2b595398c189172 SystemUUID:ec22c17f-08d8-67ba-b2b5-95398c189172 BootID:56cb277d-c51c-4bef-a9a2-659f48949c56 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ca:32:c3:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ca:32:c3:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:57:13:f2:2c:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:41:08.824731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.824730    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:41:08.824851 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.824839    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:41:08.826227 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.826193    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:41:08.826406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.826231    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:41:08.826955 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.826945    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:41:08.826996 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.826958    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:41:08.826996 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.826965    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:08.826996 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.826982    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:08.828232 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.828222    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:41:08.830083 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.830068    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:08.831980 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.831967    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:08.832106 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832096    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:41:08.832140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832126    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:41:08.832171 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832143    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:41:08.832171 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832158    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:41:08.832171 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832168    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:41:08.832264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832177    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:41:08.832754 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832740    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:08.832801 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.832765    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:08.834799 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.834783    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:41:08.837167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.837152    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:41:08.837222 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.837185    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:41:08.838344 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838332    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:41:08.838384 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838350    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:41:08.838384 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838357    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:41:08.838384 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838363    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:41:08.838384 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838371    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:41:08.838384 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838384    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:41:08.838532 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838408    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:41:08.838532 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838416    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:41:08.838532 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838422    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:41:08.838532 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838428    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:41:08.838532 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838447    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:41:08.838532 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838452    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:41:08.838934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838923    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:41:08.838966 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.838935    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:41:08.842674 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.842645    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:41:08.842968 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.842563    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:41:08.843045 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.842995    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:41:08.843813 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.843797    2585 server.go:1268] "Started kubelet"
Sep 17 17:41:08.843919 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.843870    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:41:08.844006 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.843940    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:41:08.844051 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.844026    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:41:08.844638 ip-10-0-138-144 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:41:08.845902 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.845353    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:41:08.845902 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.845390    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:41:08.848718 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.848697    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:41:08.852076 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852058    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:41:08.852076 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852069    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:08.852368 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.852306    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:08.852812 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852654    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:41:08.852812 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852671    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:41:08.852812 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852685    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:41:08.852812 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852790    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:41:08.852812 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852798    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:41:08.852812 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852806    2585 factory.go:55] Registering systemd factory
Sep 17 17:41:08.853162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.852853    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:41:08.853533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.853443    2585 factory.go:153] Registering CRI-O factory
Sep 17 17:41:08.853533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.853457    2585 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:41:08.853533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.853519    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:41:08.853688 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.853540    2585 factory.go:103] Registering Raw factory
Sep 17 17:41:08.853688 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.853556    2585 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:41:08.854000 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.853987    2585 manager.go:349] Starting recovery of all containers
Sep 17 17:41:08.855270 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.855238    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:41:08.855351 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.855334    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:41:08.862612 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.862581    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7t45s"
Sep 17 17:41:08.863285 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.863272    2585 manager.go:354] Recovery completed
Sep 17 17:41:08.865170 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.865146    2585 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 17:41:08.868124 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.868110    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:08.870361 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.870345    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:08.870440 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.870379    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:08.870440 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.870395    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:08.870982 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.870969    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:41:08.871051 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.870981    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:41:08.871051 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.871010    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:08.871165 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.871050    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:08.871216 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.871161    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7t45s"
Sep 17 17:41:08.871794 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.871769    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:08.871882 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.871802    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:08.871882 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.871817    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:08.873333 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.873319    2585 policy_none.go:49] "None policy: Start"
Sep 17 17:41:08.873414 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.873340    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:41:08.873414 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.873354    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:41:08.875622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.875606    2585 policy_none.go:47] "Start"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929515    2585 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.929554    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929566    2585 server.go:87] "Starting device plugin registration server"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929809    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929833    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929901    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929965    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.929973    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.930555    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:41:08.937743 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.930600    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:08.951841 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.951664    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:41:08.953413 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.953394    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:41:08.953481 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.953445    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:41:08.953481 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.953465    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:41:08.953562 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:08.953505    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:41:08.956229 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:08.956210    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:09.030418 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.030382    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:09.031242 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.031226    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:09.031312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.031261    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:09.031312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.031273    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:09.031312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.031297    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.038298 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.038262    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.038298 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.038282    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-144.ec2.internal\": node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.053546 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.053520    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.053625 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.053546    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"]
Sep 17 17:41:09.053625 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.053604    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:09.054467 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.054451    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:09.054536 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.054485    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:09.054536 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.054496    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:09.056730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.056715    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:09.056872 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.056858    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.056932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.056902    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:09.057403 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.057388    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:09.057487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.057414    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:09.057487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.057441    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:09.057487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.057459    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:09.057487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.057477    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:09.057487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.057486    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:09.059756 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.059740    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.059825 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.059770    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:09.060494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.060478    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:09.060593 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.060504    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:09.060593 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.060518    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:09.087646 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.087630    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-144.ec2.internal\" not found" node="ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.092272 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.092256    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-144.ec2.internal\" not found" node="ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.153662 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.153637    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.254654 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.254626    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.254735 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.254721    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf8382f69c9b1845899943c577e37580-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"bf8382f69c9b1845899943c577e37580\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.254781 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.254744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acff49d59889ccaa0427b1a9c3b732ea-config\") pod \"kube-apiserver-proxy-ip-10-0-138-144.ec2.internal\" (UID: \"acff49d59889ccaa0427b1a9c3b732ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.254781 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.254759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf8382f69c9b1845899943c577e37580-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"bf8382f69c9b1845899943c577e37580\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.355273 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.355208    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.355273 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.355249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acff49d59889ccaa0427b1a9c3b732ea-config\") pod \"kube-apiserver-proxy-ip-10-0-138-144.ec2.internal\" (UID: \"acff49d59889ccaa0427b1a9c3b732ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.355273 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.355269    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf8382f69c9b1845899943c577e37580-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"bf8382f69c9b1845899943c577e37580\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.355439 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.355284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf8382f69c9b1845899943c577e37580-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"bf8382f69c9b1845899943c577e37580\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.355439 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.355332    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf8382f69c9b1845899943c577e37580-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"bf8382f69c9b1845899943c577e37580\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.355439 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.355350    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acff49d59889ccaa0427b1a9c3b732ea-config\") pod \"kube-apiserver-proxy-ip-10-0-138-144.ec2.internal\" (UID: \"acff49d59889ccaa0427b1a9c3b732ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.355439 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.355365    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf8382f69c9b1845899943c577e37580-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"bf8382f69c9b1845899943c577e37580\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.391384 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.391364    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.395560 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.395548    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:09.455406 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.455377    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.555845 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.555790    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.656237 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.656219    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.753824 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.753788    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:09.756805 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.756787    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.800931 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.800911    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:41:09.801297 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.801017    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:09.801297 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.801051    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:09.856911 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.856883    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:09.893531 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.893499    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:36:08 +0000 UTC" deadline="2028-03-07 12:56:17.16863226 +0000 UTC"
Sep 17 17:41:09.893531 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:09.893525    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12883h15m7.275110347s"
Sep 17 17:41:09.957077 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:09.957052    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:10.029740 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:10.029711    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacff49d59889ccaa0427b1a9c3b732ea.slice/crio-d03aee936066e6c4cf4aab11c232fab62bed02397289c067fab4355c5581b66b WatchSource:0}: Error finding container d03aee936066e6c4cf4aab11c232fab62bed02397289c067fab4355c5581b66b: Status 404 returned error can't find the container with id d03aee936066e6c4cf4aab11c232fab62bed02397289c067fab4355c5581b66b
Sep 17 17:41:10.034338 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.034323    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:10.053538 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:10.053518    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8382f69c9b1845899943c577e37580.slice/crio-7eebb7e140addde7991ff3c617cecb19e4ebd2a3ebb83fd575e28599dd531986 WatchSource:0}: Error finding container 7eebb7e140addde7991ff3c617cecb19e4ebd2a3ebb83fd575e28599dd531986: Status 404 returned error can't find the container with id 7eebb7e140addde7991ff3c617cecb19e4ebd2a3ebb83fd575e28599dd531986
Sep 17 17:41:10.057853 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.057797    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:10.158413 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.158378    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Sep 17 17:41:10.177630 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.177585    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:10.253270 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.253244    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:10.265102 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.265083    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:10.266549 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.266536    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Sep 17 17:41:10.276058 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.276043    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:10.276379 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.276363    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:10.720491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.720451    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:10.833276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.833238    2585 apiserver.go:52] "Watching apiserver"
Sep 17 17:41:10.842963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.842933    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:41:10.843681 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.843649    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-29686","openshift-network-diagnostics/network-check-target-kt64l","openshift-network-operator/iptables-alerter-7pvw9","openshift-ovn-kubernetes/ovnkube-node-g9d4s","kube-system/konnectivity-agent-pr9b6","kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp","openshift-cluster-node-tuning-operator/tuned-j5nt8","openshift-image-registry/node-ca-kggnz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal","openshift-multus/multus-additional-cni-plugins-sbbqb","openshift-multus/multus-phph5","kube-system/global-pull-secret-syncer-6g6jp","openshift-dns/node-resolver-nvkjm"]
Sep 17 17:41:10.846268 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.846248    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.848515 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.848492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.848611 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.848585    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:10.849519 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.849499    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b9l7v\""
Sep 17 17:41:10.849848 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.849824    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:41:10.849966 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.849858    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.849966 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.849959    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.850487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.850465    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.852573 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.852555    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:10.853134 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.853115    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.853662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.853630    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.854140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.854124    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:41:10.854226 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.854127    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.855538 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.855520    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vc5qt\""
Sep 17 17:41:10.856568 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.856481    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:10.857253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.856692    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:41:10.857253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.856913    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:41:10.857253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.857045    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.857253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.857047    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:41:10.857253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.857165    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.857917 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.857895    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-45qgh\""
Sep 17 17:41:10.858147 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.858118    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:41:10.859079 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.859060    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cd75v\""
Sep 17 17:41:10.859365 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.859347    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:41:10.859921 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.859701    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:41:10.860252 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.860151    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:10.862569 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.862549    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:10.863199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.863178    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.863286 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.863198    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9rw7\""
Sep 17 17:41:10.864530 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864507    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64ff\" (UniqueName: \"kubernetes.io/projected/4f77808b-f0e6-4982-8dc4-6839a9fca62b-kube-api-access-t64ff\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.864619 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-slash\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864619 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864570    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovnkube-script-lib\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864619 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864606    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864767 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-ovn\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864767 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864698    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-sys-fs\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.864767 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94ch\" (UniqueName: \"kubernetes.io/projected/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-kube-api-access-m94ch\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.864932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864788    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-run-netns\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-etc-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864865    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864888    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovnkube-config\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.864932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864911    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovn-node-metrics-cert\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864933    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864956    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-socket-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.864988    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865022    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f77808b-f0e6-4982-8dc4-6839a9fca62b-iptables-alerter-script\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-kubelet\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-systemd-units\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865130    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-cni-netd\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865170 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865154    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865176    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-dbus\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865195    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f77808b-f0e6-4982-8dc4-6839a9fca62b-host-slash\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865213    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-log-socket\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865232    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.865297    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-env-overrides\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-registration-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-device-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.865399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865393    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-var-lib-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865413    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-node-log\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9429q\" (UniqueName: \"kubernetes.io/projected/3075df7b-db2c-4495-8264-1a9102b2fe4b-kube-api-access-9429q\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865457    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.865670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865479    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-kubelet-config\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.865670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-systemd\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.865670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.865512    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-cni-bin\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.866687 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.866167    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:41:10.866687 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.866213    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.866687 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.866408    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.866925 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.866903    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.867036 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.866952    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sxwd6\""
Sep 17 17:41:10.869297 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.869081    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:10.869297 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.869185    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:10.872286 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.872268    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.874915 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.874897    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.877462 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.877352    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.885147 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.885126    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.885397 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.885380    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:41:10.885585 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.885567    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:41:10.885631 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.885599    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tt2nm\""
Sep 17 17:41:10.885709 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.885698    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:41:10.885804 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.885788    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.890204 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.890183    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:41:10.890451 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.890432    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:41:10.890562 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.890545    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:41:10.890920 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.890766    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwc8b\""
Sep 17 17:41:10.890920 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.890802    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:10.891134 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.891116    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9t6xf\""
Sep 17 17:41:10.901310 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.901285    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cp6ln"
Sep 17 17:41:10.908929 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.908907    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cp6ln"
Sep 17 17:41:10.953968 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.953943    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:41:10.958455 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.958406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" event={"ID":"bf8382f69c9b1845899943c577e37580","Type":"ContainerStarted","Data":"7eebb7e140addde7991ff3c617cecb19e4ebd2a3ebb83fd575e28599dd531986"}
Sep 17 17:41:10.959966 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.959948    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal" event={"ID":"acff49d59889ccaa0427b1a9c3b732ea","Type":"ContainerStarted","Data":"d03aee936066e6c4cf4aab11c232fab62bed02397289c067fab4355c5581b66b"}
Sep 17 17:41:10.966251 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966227    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-var-lib-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966377 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966260    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-node-log\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966377 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9429q\" (UniqueName: \"kubernetes.io/projected/3075df7b-db2c-4495-8264-1a9102b2fe4b-kube-api-access-9429q\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966377 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966340    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-daemon-config\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.966377 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966355    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-var-lib-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966377 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966365    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-systemd\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-host\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966430    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966491    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dqx\" (UniqueName: \"kubernetes.io/projected/2177cca8-a58d-4b81-8a10-65f67ee8ada0-kube-api-access-q9dqx\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966496    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-node-log\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966541    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64ff\" (UniqueName: \"kubernetes.io/projected/4f77808b-f0e6-4982-8dc4-6839a9fca62b-kube-api-access-t64ff\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966563    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovnkube-script-lib\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966586    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-modprobe-d\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966607    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-run\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-cni-multus\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966652    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwts5\" (UniqueName: \"kubernetes.io/projected/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-kube-api-access-xwts5\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-ovn\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94ch\" (UniqueName: \"kubernetes.io/projected/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-kube-api-access-m94ch\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cnibin\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966799    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-ovn\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966800    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966834    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-etc-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.966853 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966872    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovnkube-config\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysctl-d\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966914    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-cni-bin\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966939    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966956    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966964    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-k8s-cni-cncf-io\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.966999    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f77808b-f0e6-4982-8dc4-6839a9fca62b-iptables-alerter-script\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967008    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-etc-openvswitch\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967023    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-systemd-units\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967048    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysconfig\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-kubernetes\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967106    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-systemd-units\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967114    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-var-lib-kubelet\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967140    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbqhk\" (UniqueName: \"kubernetes.io/projected/b1601798-247c-4826-b3ba-c2b0b26ab562-kube-api-access-qbqhk\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-cni-netd\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967184    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.967348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967195    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovnkube-script-lib\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-dbus\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967239    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-cnibin\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-cni-binary-copy\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967284    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-kubelet\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-etc-kubernetes\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967334    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967359    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-env-overrides\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967380    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-registration-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967400    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-lib-modules\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967418    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/750bb16e-a510-490e-8b0f-285c9ea415ac-agent-certs\") pod \"konnectivity-agent-pr9b6\" (UID: \"750bb16e-a510-490e-8b0f-285c9ea415ac\") " pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2177cca8-a58d-4b81-8a10-65f67ee8ada0-host\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-dbus\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-kubelet-config\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967510    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovnkube-config\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2177cca8-a58d-4b81-8a10-65f67ee8ada0-serviceca\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:10.968144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967513    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-cni-netd\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967465    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-systemd\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967628    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967646    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-registration-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967647    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f77808b-f0e6-4982-8dc4-6839a9fca62b-iptables-alerter-script\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-cni-bin\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967674    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-kubelet-config\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967702    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-sys\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967710    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-run-systemd\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-cni-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967771    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-cni-bin\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdk86\" (UniqueName: \"kubernetes.io/projected/9e6f7109-7114-4a3e-922f-408eb432d5dd-kube-api-access-zdk86\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967850    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3075df7b-db2c-4495-8264-1a9102b2fe4b-env-overrides\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-slash\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-slash\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.968963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967950    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysctl-conf\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.967988    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968012    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjcc\" (UniqueName: \"kubernetes.io/projected/db93ed24-9b8e-49c4-96a4-164a1d955bb5-kube-api-access-pnjcc\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968038    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/750bb16e-a510-490e-8b0f-285c9ea415ac-konnectivity-ca\") pod \"konnectivity-agent-pr9b6\" (UID: \"750bb16e-a510-490e-8b0f-285c9ea415ac\") " pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968068    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968110    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnr5v\" (UniqueName: \"kubernetes.io/projected/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-kube-api-access-bnr5v\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-sys-fs\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968161    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-tuned\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-socket-dir-parent\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968210    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-run-netns\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968239    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-sys-fs\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968244    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovn-node-metrics-cert\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968272    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-run-netns\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968335    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-socket-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:10.969544 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968512    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-netns\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968519    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-socket-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.968548    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968548    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-conf-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968579    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6f7109-7114-4a3e-922f-408eb432d5dd-tmp-dir\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968588    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968606    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-kubelet\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:10.968645    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.468607082 +0000 UTC m=+4.659944765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968651    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-host-kubelet\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1601798-247c-4826-b3ba-c2b0b26ab562-tmp\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968701    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-multus-certs\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968726    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e6f7109-7114-4a3e-922f-408eb432d5dd-hosts-file\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-system-cni-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-hostroot\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f77808b-f0e6-4982-8dc4-6839a9fca62b-host-slash\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-log-socket\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.970163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968906    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f77808b-f0e6-4982-8dc4-6839a9fca62b-host-slash\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.970732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-device-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.970732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.968957    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:10.970732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.969007    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-device-dir\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.970732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.969035    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3075df7b-db2c-4495-8264-1a9102b2fe4b-log-socket\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.972551 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.972530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3075df7b-db2c-4495-8264-1a9102b2fe4b-ovn-node-metrics-cert\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:10.976233 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.976213    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64ff\" (UniqueName: \"kubernetes.io/projected/4f77808b-f0e6-4982-8dc4-6839a9fca62b-kube-api-access-t64ff\") pod \"iptables-alerter-7pvw9\" (UID: \"4f77808b-f0e6-4982-8dc4-6839a9fca62b\") " pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:10.976531 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.976513    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94ch\" (UniqueName: \"kubernetes.io/projected/a4885a76-c4ed-4eff-bf0e-424be1bb8a56-kube-api-access-m94ch\") pod \"aws-ebs-csi-driver-node-zhrbp\" (UID: \"a4885a76-c4ed-4eff-bf0e-424be1bb8a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:10.976623 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:10.976598    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9429q\" (UniqueName: \"kubernetes.io/projected/3075df7b-db2c-4495-8264-1a9102b2fe4b-kube-api-access-9429q\") pod \"ovnkube-node-g9d4s\" (UID: \"3075df7b-db2c-4495-8264-1a9102b2fe4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:11.069924 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.069890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysctl-d\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.069934    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-cni-bin\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.069961    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.070073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.069986    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-k8s-cni-cncf-io\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070027    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysconfig\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070027    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-cni-bin\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070063    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-kubernetes\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-var-lib-kubelet\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070095    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysctl-d\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070110    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbqhk\" (UniqueName: \"kubernetes.io/projected/b1601798-247c-4826-b3ba-c2b0b26ab562-kube-api-access-qbqhk\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-k8s-cni-cncf-io\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070121    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysconfig\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-kubernetes\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070137    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-cnibin\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070165    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-var-lib-kubelet\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-cni-binary-copy\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070215    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-kubelet\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-cnibin\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.070422 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070321    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-kubelet\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.071222 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.070667    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-etc-kubernetes\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.071222 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.071796 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071768    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-cni-binary-copy\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.071912 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071846    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-etc-kubernetes\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.071912 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071856    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:11.071981 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071927    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-lib-modules\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.071981 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071957    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/750bb16e-a510-490e-8b0f-285c9ea415ac-agent-certs\") pod \"konnectivity-agent-pr9b6\" (UID: \"750bb16e-a510-490e-8b0f-285c9ea415ac\") " pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:11.072042 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.071984    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2177cca8-a58d-4b81-8a10-65f67ee8ada0-host\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.072042 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072011    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2177cca8-a58d-4b81-8a10-65f67ee8ada0-serviceca\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.072107 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-sys\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072107 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-cni-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072107 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdk86\" (UniqueName: \"kubernetes.io/projected/9e6f7109-7114-4a3e-922f-408eb432d5dd-kube-api-access-zdk86\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.072190 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072124    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.072190 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysctl-conf\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072190 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.072294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072201    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjcc\" (UniqueName: \"kubernetes.io/projected/db93ed24-9b8e-49c4-96a4-164a1d955bb5-kube-api-access-pnjcc\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.072294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072230    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/750bb16e-a510-490e-8b0f-285c9ea415ac-konnectivity-ca\") pod \"konnectivity-agent-pr9b6\" (UID: \"750bb16e-a510-490e-8b0f-285c9ea415ac\") " pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:11.072294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:11.072294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnr5v\" (UniqueName: \"kubernetes.io/projected/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-kube-api-access-bnr5v\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:11.072406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072324    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-tuned\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072350    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-socket-dir-parent\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-netns\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072490 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-conf-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072490 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6f7109-7114-4a3e-922f-408eb432d5dd-tmp-dir\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.072490 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072478    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1601798-247c-4826-b3ba-c2b0b26ab562-tmp\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072576 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072505    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-multus-certs\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072576 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e6f7109-7114-4a3e-922f-408eb432d5dd-hosts-file\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.072576 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072554    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-system-cni-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072581    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-hostroot\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.072662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072647    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-daemon-config\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072672    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-systemd\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-host\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072743 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072722    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.072839 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072752    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dqx\" (UniqueName: \"kubernetes.io/projected/2177cca8-a58d-4b81-8a10-65f67ee8ada0-kube-api-access-q9dqx\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.072839 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072794    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-modprobe-d\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072840    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-run\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.072940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-cni-multus\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072892    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwts5\" (UniqueName: \"kubernetes.io/projected/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-kube-api-access-xwts5\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072914    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-netns\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.072940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072923    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cnibin\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.073083 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.072980    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-lib-modules\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.073083 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073029    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-conf-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.073083 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073046    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-host\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.073083 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073077    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.073404 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073342    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-cni-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.073404 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073383    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6f7109-7114-4a3e-922f-408eb432d5dd-tmp-dir\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.073556 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-socket-dir-parent\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073675    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2177cca8-a58d-4b81-8a10-65f67ee8ada0-host\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073783    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-modprobe-d\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073844    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-run\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073877    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-system-cni-dir\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073884    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-var-lib-cni-multus\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.073932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cnibin\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.074436    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.074588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db93ed24-9b8e-49c4-96a4-164a1d955bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.074602    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/750bb16e-a510-490e-8b0f-285c9ea415ac-konnectivity-ca\") pod \"konnectivity-agent-pr9b6\" (UID: \"750bb16e-a510-490e-8b0f-285c9ea415ac\") " pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.074715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-sysctl-conf\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.074976    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2177cca8-a58d-4b81-8a10-65f67ee8ada0-serviceca\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075032    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-sys\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075190    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db93ed24-9b8e-49c4-96a4-164a1d955bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075252    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-hostroot\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.075276 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.075282    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:11.076088 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-systemd\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.076088 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075357    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e6f7109-7114-4a3e-922f-408eb432d5dd-hosts-file\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.076088 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.075396    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.575375895 +0000 UTC m=+4.766713572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:11.076088 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075522    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/750bb16e-a510-490e-8b0f-285c9ea415ac-agent-certs\") pod \"konnectivity-agent-pr9b6\" (UID: \"750bb16e-a510-490e-8b0f-285c9ea415ac\") " pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:11.076088 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-host-run-multus-certs\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.076088 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.075893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-multus-daemon-config\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.076514 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.076492    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1601798-247c-4826-b3ba-c2b0b26ab562-tmp\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.078536 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.078515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1601798-247c-4826-b3ba-c2b0b26ab562-etc-tuned\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.081731 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.081714    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:11.081856 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.081734    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:11.081856 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.081747    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:11.081856 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.081793    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.581779647 +0000 UTC m=+4.773117314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:11.083639 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.083614    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dqx\" (UniqueName: \"kubernetes.io/projected/2177cca8-a58d-4b81-8a10-65f67ee8ada0-kube-api-access-q9dqx\") pod \"node-ca-kggnz\" (UID: \"2177cca8-a58d-4b81-8a10-65f67ee8ada0\") " pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.083945 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.083910    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjcc\" (UniqueName: \"kubernetes.io/projected/db93ed24-9b8e-49c4-96a4-164a1d955bb5-kube-api-access-pnjcc\") pod \"multus-additional-cni-plugins-sbbqb\" (UID: \"db93ed24-9b8e-49c4-96a4-164a1d955bb5\") " pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.084033 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.084007    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdk86\" (UniqueName: \"kubernetes.io/projected/9e6f7109-7114-4a3e-922f-408eb432d5dd-kube-api-access-zdk86\") pod \"node-resolver-nvkjm\" (UID: \"9e6f7109-7114-4a3e-922f-408eb432d5dd\") " pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.084434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.084415    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwts5\" (UniqueName: \"kubernetes.io/projected/19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c-kube-api-access-xwts5\") pod \"multus-phph5\" (UID: \"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c\") " pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.084511 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.084466    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbqhk\" (UniqueName: \"kubernetes.io/projected/b1601798-247c-4826-b3ba-c2b0b26ab562-kube-api-access-qbqhk\") pod \"tuned-j5nt8\" (UID: \"b1601798-247c-4826-b3ba-c2b0b26ab562\") " pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.085173 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.085152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnr5v\" (UniqueName: \"kubernetes.io/projected/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-kube-api-access-bnr5v\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:11.158547 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.158516    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp"
Sep 17 17:41:11.166755 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.166719    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7pvw9"
Sep 17 17:41:11.175510 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.175490    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:11.182984 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.182964    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:11.196611 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.196593    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nvkjm"
Sep 17 17:41:11.202024 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.202007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kggnz"
Sep 17 17:41:11.210159 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.210143    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j5nt8"
Sep 17 17:41:11.217370 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.217346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sbbqb"
Sep 17 17:41:11.224870 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.224827    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phph5"
Sep 17 17:41:11.475055 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.474978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:11.475176 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.475102    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:11.475176 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.475165    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:12.475152154 +0000 UTC m=+5.666489830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:11.575779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.575743    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:11.575965 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.575943    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:11.576026 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.576014    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:12.575997735 +0000 UTC m=+5.767335402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:11.676068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.676037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:11.676253 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.676230    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:11.676318 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.676262    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:11.676318 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.676276    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:11.676420 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:11.676343    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:12.676324316 +0000 UTC m=+5.867661994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:11.877158 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.877126    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750bb16e_a510_490e_8b0f_285c9ea415ac.slice/crio-3f9434ee0a1eefaf82923cd6dd21f51f00475b8bd6f5f82942ea8bcf6e2f23c2 WatchSource:0}: Error finding container 3f9434ee0a1eefaf82923cd6dd21f51f00475b8bd6f5f82942ea8bcf6e2f23c2: Status 404 returned error can't find the container with id 3f9434ee0a1eefaf82923cd6dd21f51f00475b8bd6f5f82942ea8bcf6e2f23c2
Sep 17 17:41:11.879010 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.878766    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb93ed24_9b8e_49c4_96a4_164a1d955bb5.slice/crio-e89532f28b2cbbf6dd710afe657f41ec4006d468f58c9120055d8cd705f012de WatchSource:0}: Error finding container e89532f28b2cbbf6dd710afe657f41ec4006d468f58c9120055d8cd705f012de: Status 404 returned error can't find the container with id e89532f28b2cbbf6dd710afe657f41ec4006d468f58c9120055d8cd705f012de
Sep 17 17:41:11.882551 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.882528    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1601798_247c_4826_b3ba_c2b0b26ab562.slice/crio-266634e8ddf306befde97e0fa55102f206297bc4ece0839d323b7f5f81581e69 WatchSource:0}: Error finding container 266634e8ddf306befde97e0fa55102f206297bc4ece0839d323b7f5f81581e69: Status 404 returned error can't find the container with id 266634e8ddf306befde97e0fa55102f206297bc4ece0839d323b7f5f81581e69
Sep 17 17:41:11.883285 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.883262    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f7109_7114_4a3e_922f_408eb432d5dd.slice/crio-4e427cf7c99079a1e041157f53f10c8a1d4ddcd5370bb05b3abd45f3b665f73c WatchSource:0}: Error finding container 4e427cf7c99079a1e041157f53f10c8a1d4ddcd5370bb05b3abd45f3b665f73c: Status 404 returned error can't find the container with id 4e427cf7c99079a1e041157f53f10c8a1d4ddcd5370bb05b3abd45f3b665f73c
Sep 17 17:41:11.885126 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.884883    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4885a76_c4ed_4eff_bf0e_424be1bb8a56.slice/crio-b24f5039523f74d4e3d6bdb40f553fc7b83b2e7d0940a251b16202f8da1d50ce WatchSource:0}: Error finding container b24f5039523f74d4e3d6bdb40f553fc7b83b2e7d0940a251b16202f8da1d50ce: Status 404 returned error can't find the container with id b24f5039523f74d4e3d6bdb40f553fc7b83b2e7d0940a251b16202f8da1d50ce
Sep 17 17:41:11.885625 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.885602    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f77808b_f0e6_4982_8dc4_6839a9fca62b.slice/crio-cdc904c77435d20806508e8a9bc1b062abb01381919828e1212834bafd7cac49 WatchSource:0}: Error finding container cdc904c77435d20806508e8a9bc1b062abb01381919828e1212834bafd7cac49: Status 404 returned error can't find the container with id cdc904c77435d20806508e8a9bc1b062abb01381919828e1212834bafd7cac49
Sep 17 17:41:11.886643 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.886621    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e0e20a_4fbb_4b22_bfc0_d08736b8fd4c.slice/crio-d18023c1eb3d272c7845369fb1484d231ffd0e52bbce076cb58a9162792532b0 WatchSource:0}: Error finding container d18023c1eb3d272c7845369fb1484d231ffd0e52bbce076cb58a9162792532b0: Status 404 returned error can't find the container with id d18023c1eb3d272c7845369fb1484d231ffd0e52bbce076cb58a9162792532b0
Sep 17 17:41:11.887220 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.887142    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3075df7b_db2c_4495_8264_1a9102b2fe4b.slice/crio-ce21678bd763ad43d072397634b97efb267788f71e7967cc0cb8db148afcf15f WatchSource:0}: Error finding container ce21678bd763ad43d072397634b97efb267788f71e7967cc0cb8db148afcf15f: Status 404 returned error can't find the container with id ce21678bd763ad43d072397634b97efb267788f71e7967cc0cb8db148afcf15f
Sep 17 17:41:11.908775 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:11.908752    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2177cca8_a58d_4b81_8a10_65f67ee8ada0.slice/crio-4e545f4bc3ee56c9e1e796d1582f332d198af9d321a76a8785fa37aa86bd8325 WatchSource:0}: Error finding container 4e545f4bc3ee56c9e1e796d1582f332d198af9d321a76a8785fa37aa86bd8325: Status 404 returned error can't find the container with id 4e545f4bc3ee56c9e1e796d1582f332d198af9d321a76a8785fa37aa86bd8325
Sep 17 17:41:11.911782 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.911754    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:10 +0000 UTC" deadline="2028-05-27 01:39:48.479208938 +0000 UTC"
Sep 17 17:41:11.911872 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.911788    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14815h58m36.567425745s"
Sep 17 17:41:11.963451 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.963422    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp" event={"ID":"a4885a76-c4ed-4eff-bf0e-424be1bb8a56","Type":"ContainerStarted","Data":"b24f5039523f74d4e3d6bdb40f553fc7b83b2e7d0940a251b16202f8da1d50ce"}
Sep 17 17:41:11.964538 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.964512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pr9b6" event={"ID":"750bb16e-a510-490e-8b0f-285c9ea415ac","Type":"ContainerStarted","Data":"3f9434ee0a1eefaf82923cd6dd21f51f00475b8bd6f5f82942ea8bcf6e2f23c2"}
Sep 17 17:41:11.965636 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.965589    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phph5" event={"ID":"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c","Type":"ContainerStarted","Data":"d18023c1eb3d272c7845369fb1484d231ffd0e52bbce076cb58a9162792532b0"}
Sep 17 17:41:11.966587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.966563    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nvkjm" event={"ID":"9e6f7109-7114-4a3e-922f-408eb432d5dd","Type":"ContainerStarted","Data":"4e427cf7c99079a1e041157f53f10c8a1d4ddcd5370bb05b3abd45f3b665f73c"}
Sep 17 17:41:11.967575 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.967555    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j5nt8" event={"ID":"b1601798-247c-4826-b3ba-c2b0b26ab562","Type":"ContainerStarted","Data":"266634e8ddf306befde97e0fa55102f206297bc4ece0839d323b7f5f81581e69"}
Sep 17 17:41:11.968782 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.968759    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerStarted","Data":"e89532f28b2cbbf6dd710afe657f41ec4006d468f58c9120055d8cd705f012de"}
Sep 17 17:41:11.970014 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.969981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kggnz" event={"ID":"2177cca8-a58d-4b81-8a10-65f67ee8ada0","Type":"ContainerStarted","Data":"4e545f4bc3ee56c9e1e796d1582f332d198af9d321a76a8785fa37aa86bd8325"}
Sep 17 17:41:11.971169 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.971147    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"ce21678bd763ad43d072397634b97efb267788f71e7967cc0cb8db148afcf15f"}
Sep 17 17:41:11.972236 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:11.972216    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7pvw9" event={"ID":"4f77808b-f0e6-4982-8dc4-6839a9fca62b","Type":"ContainerStarted","Data":"cdc904c77435d20806508e8a9bc1b062abb01381919828e1212834bafd7cac49"}
Sep 17 17:41:12.481014 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.480936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:12.481167 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.481077    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:12.481167 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.481138    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:14.481120493 +0000 UTC m=+7.672458160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:12.585830 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.582269    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:12.585830 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.582413    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:12.585830 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.582474    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:14.582455084 +0000 UTC m=+7.773792764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:12.671342 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.671127    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:12.683665 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.683102    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:12.683665 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.683282    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:12.683665 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.683303    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:12.683665 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.683317    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:12.683665 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.683368    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:14.683351394 +0000 UTC m=+7.874689072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:12.912253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.912202    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:10 +0000 UTC" deadline="2028-04-02 01:07:31.946890755 +0000 UTC"
Sep 17 17:41:12.912253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.912250    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13495h26m19.034644792s"
Sep 17 17:41:12.956917 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.956870    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:12.957082 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.957008    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:12.957410 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.957386    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:12.957506 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.957489    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:12.957565 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.957550    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:12.957795 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:12.957615    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:12.984388 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.982991    2585 generic.go:356] "Generic (PLEG): container finished" podID="bf8382f69c9b1845899943c577e37580" containerID="31f631dac924b596fa6023fd0b36a35853bd8d55cdabef9da90e06f511fa4b8a" exitCode=0
Sep 17 17:41:12.984388 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:12.983071    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" event={"ID":"bf8382f69c9b1845899943c577e37580","Type":"ContainerDied","Data":"31f631dac924b596fa6023fd0b36a35853bd8d55cdabef9da90e06f511fa4b8a"}
Sep 17 17:41:13.003592 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:13.003196    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal" event={"ID":"acff49d59889ccaa0427b1a9c3b732ea","Type":"ContainerStarted","Data":"a0223ba59f920493d1627b4f66a4c7ddaea280912b861ae0997220b679dceb63"}
Sep 17 17:41:14.019772 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.019739    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" event={"ID":"bf8382f69c9b1845899943c577e37580","Type":"ContainerStarted","Data":"c9b5f2aa0486ec40e45ed56c60e72bad0f062e4c4421a0ec619ce1c89d500a04"}
Sep 17 17:41:14.035389 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.035335    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal" podStartSLOduration=4.035313448 podStartE2EDuration="4.035313448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:13.021071063 +0000 UTC m=+6.212408749" watchObservedRunningTime="2026-09-17 17:41:14.035313448 +0000 UTC m=+7.226651124"
Sep 17 17:41:14.510200 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.509681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:14.510200 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.509833    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:14.510200 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.509890    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:18.509872516 +0000 UTC m=+11.701210186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:14.610657 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.610134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:14.610657 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.610286    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:14.610657 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.610339    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:18.610323118 +0000 UTC m=+11.801660797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:14.710633 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.710593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:14.710883 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.710861    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:14.710958 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.710887    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:14.710958 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.710899    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:14.710958 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.710957    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:18.710939542 +0000 UTC m=+11.902277218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:14.958346 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.957624    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:14.958346 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.957737    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:14.958346 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.957826    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:14.958346 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.957897    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:14.958346 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:14.957951    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:14.958346 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:14.958030    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:16.955313 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:16.954656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:16.955313 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:16.954790    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:16.955313 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:16.955168    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:16.955313 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:16.955262    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:16.956038 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:16.956008    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:16.956154 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:16.956108    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:18.541483 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:18.541442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:18.541932 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.541578    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:18.541932 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.541644    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.541626062 +0000 UTC m=+19.732963730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:18.642523 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:18.642474    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:18.642674 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.642600    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:18.642731 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.642664    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.642646311 +0000 UTC m=+19.833983973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:18.743629 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:18.743593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:18.743842 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.743734    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:18.743842 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.743752    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:18.743842 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.743765    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:18.743842 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.743827    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.743798325 +0000 UTC m=+19.935135992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:18.954980 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:18.954945    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:18.955151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:18.954947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:18.955151 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.955056    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:18.955151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:18.955068    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:18.955309 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.955165    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:18.955309 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:18.955246    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:20.954034 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:20.954004    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:20.954468 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:20.954056    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:20.954468 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:20.954074    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:20.954468 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:20.954183    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:20.954468 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:20.954260    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:20.954468 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:20.954365    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:22.954233 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:22.954198    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:22.954673 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:22.954198    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:22.954673 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:22.954330    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:22.954673 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:22.954346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:22.954673 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:22.954448    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:22.954673 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:22.954542    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:24.954057 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:24.954024    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:24.954497 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:24.954035    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:24.954497 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:24.954151    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:24.954497 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:24.954032    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:24.954497 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:24.954234    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:24.954497 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:24.954338    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:26.599000 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:26.598962    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:26.599427 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.599104    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:26.599427 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.599169    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.5991492 +0000 UTC m=+35.790486877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:26.699474 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:26.699445    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:26.699622 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.699600    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:26.699668 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.699659    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.699643985 +0000 UTC m=+35.890981671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:26.800059 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:26.800026    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:26.800221 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.800195    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:26.800221 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.800217    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:26.800300 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.800229    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:26.800300 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.800283    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.800270267 +0000 UTC m=+35.991607933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:26.954530 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:26.954497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:26.954701 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.954597    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:26.954701 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:26.954622    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:26.954701 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:26.954506    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:26.954838 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.954726    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:26.954879 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:26.954840    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:28.955411 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:28.955380    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:28.955411 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:28.955409    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:28.955848 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:28.955486    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:28.955848 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:28.955557    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:28.955848 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:28.955608    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:28.955848 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:28.955660    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:30.053704 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.053490    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerStarted","Data":"0ada9149e48fa08bc71dc1bad1ef7b0d716d72f8efa2d9227cbd6dcd1a65c748"}
Sep 17 17:41:30.054983 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.054962    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kggnz" event={"ID":"2177cca8-a58d-4b81-8a10-65f67ee8ada0","Type":"ContainerStarted","Data":"000dedb62042006bf5b9761a52838cd9de0668665eae35e468eb52ae3fded6ac"}
Sep 17 17:41:30.056500 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.056482    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"8e641bf3f825dd7fb2ff0514cd960eb92e2bcfab2c3e865eab8d9050dde82c30"}
Sep 17 17:41:30.056581 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.056540    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"7f615c445a0d37c5df903ff4d7f5102e8149ceb22105056d880116e9fab4ce5c"}
Sep 17 17:41:30.057680 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.057663    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp" event={"ID":"a4885a76-c4ed-4eff-bf0e-424be1bb8a56","Type":"ContainerStarted","Data":"a456c2fbccc87a9855abc16b7c5bd238cff00ecff030b2e29ca8f7ac8ecc250f"}
Sep 17 17:41:30.059172 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.059151    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pr9b6" event={"ID":"750bb16e-a510-490e-8b0f-285c9ea415ac","Type":"ContainerStarted","Data":"cb6f7b3e573ba344b3815bf26b467f02614054bf840a64f4e173a908fe2ce298"}
Sep 17 17:41:30.060524 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.060504    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phph5" event={"ID":"19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c","Type":"ContainerStarted","Data":"b1481c2f735111ec95de96cab19a16c37cd3de95b288d4bc22aae7a7e9bc40b8"}
Sep 17 17:41:30.061759 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.061733    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nvkjm" event={"ID":"9e6f7109-7114-4a3e-922f-408eb432d5dd","Type":"ContainerStarted","Data":"063714d6ed1f8f16ab9291e374e5ec7e7b03c89b0530e611c95d432b36f95ffe"}
Sep 17 17:41:30.063509 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.063486    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j5nt8" event={"ID":"b1601798-247c-4826-b3ba-c2b0b26ab562","Type":"ContainerStarted","Data":"d2a3f82845591dc46f625a1b2fe73a024cd7837bbde75746644c41fa72da7840"}
Sep 17 17:41:30.081498 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.081465    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" podStartSLOduration=20.081454664 podStartE2EDuration="20.081454664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:14.036608604 +0000 UTC m=+7.227946290" watchObservedRunningTime="2026-09-17 17:41:30.081454664 +0000 UTC m=+23.272792354"
Sep 17 17:41:30.095802 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.095768    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nvkjm" podStartSLOduration=3.387094551 podStartE2EDuration="21.095753021s" totalImagesPullingTime="17.70865847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.095562789 +0000 UTC m=+23.286900473" watchObservedRunningTime="2026-09-17 17:41:30.095753021 +0000 UTC m=+23.287090707"
Sep 17 17:41:30.132421 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.132371    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j5nt8" podStartSLOduration=3.421714934 podStartE2EDuration="21.132352573s" totalImagesPullingTime="17.710637639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.115797101 +0000 UTC m=+23.307134786" watchObservedRunningTime="2026-09-17 17:41:30.132352573 +0000 UTC m=+23.323690257"
Sep 17 17:41:30.132597 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.132474    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kggnz" podStartSLOduration=3.449790945 podStartE2EDuration="21.132469313s" totalImagesPullingTime="17.682678368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.132072078 +0000 UTC m=+23.323409767" watchObservedRunningTime="2026-09-17 17:41:30.132469313 +0000 UTC m=+23.323806997"
Sep 17 17:41:30.150825 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.150776    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-phph5" podStartSLOduration=3.420903244 podStartE2EDuration="21.150765379s" totalImagesPullingTime="17.729862135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.150665084 +0000 UTC m=+23.342002770" watchObservedRunningTime="2026-09-17 17:41:30.150765379 +0000 UTC m=+23.342103064"
Sep 17 17:41:30.793174 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.792939    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:41:30.938841 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.938706    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:30.793174785Z","UUID":"a342e580-5e6e-44fb-a076-0638b0cfb988","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:41:30.940007 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.939991    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:41:30.940074 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.940017    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:41:30.954156 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.954139    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:30.954233 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.954145    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:30.954285 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:30.954227    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:30.954321 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:30.954145    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:30.954321 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:30.954307    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:30.954448 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:30.954428    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:31.068222 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068203    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:41:31.068779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068526    2585 generic.go:356] "Generic (PLEG): container finished" podID="3075df7b-db2c-4495-8264-1a9102b2fe4b" containerID="8e641bf3f825dd7fb2ff0514cd960eb92e2bcfab2c3e865eab8d9050dde82c30" exitCode=1
Sep 17 17:41:31.068779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068578    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerDied","Data":"8e641bf3f825dd7fb2ff0514cd960eb92e2bcfab2c3e865eab8d9050dde82c30"}
Sep 17 17:41:31.068779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068636    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"02829b0c534a9ee4550929ce160579279f8355f30dc3f1ff2e38796f51324ea5"}
Sep 17 17:41:31.068779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"24053706c3d017d614cad3329fd373854e07615184d3b87efba5b08def5dc9fd"}
Sep 17 17:41:31.068779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068693    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"444532dd1b07626d0bb7b7cee257d5adefcfd582a501a800c2cc3a48951d4564"}
Sep 17 17:41:31.068779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.068724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"78c36ad1f4bb0d4a8e4b2f84fe1f79a052ed759789e8c0537c43114d16840785"}
Sep 17 17:41:31.069975 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.069951    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7pvw9" event={"ID":"4f77808b-f0e6-4982-8dc4-6839a9fca62b","Type":"ContainerStarted","Data":"5ce2e6c455cc6dd744672ba3a145f1b4c1a14249b7beac01de0faaa5fa1d6866"}
Sep 17 17:41:31.071749 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.071728    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp" event={"ID":"a4885a76-c4ed-4eff-bf0e-424be1bb8a56","Type":"ContainerStarted","Data":"93001ada6f50ead3e783b517ffa7aff59a7b3e1e2d6c020d014fdd950d9978cd"}
Sep 17 17:41:31.073126 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.073109    2585 generic.go:356] "Generic (PLEG): container finished" podID="db93ed24-9b8e-49c4-96a4-164a1d955bb5" containerID="0ada9149e48fa08bc71dc1bad1ef7b0d716d72f8efa2d9227cbd6dcd1a65c748" exitCode=0
Sep 17 17:41:31.073213 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.073192    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerDied","Data":"0ada9149e48fa08bc71dc1bad1ef7b0d716d72f8efa2d9227cbd6dcd1a65c748"}
Sep 17 17:41:31.089850 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.089789    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7pvw9" podStartSLOduration=4.402159831 podStartE2EDuration="22.089778587s" totalImagesPullingTime="17.687618756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.08950613 +0000 UTC m=+24.280843827" watchObservedRunningTime="2026-09-17 17:41:31.089778587 +0000 UTC m=+24.281116269"
Sep 17 17:41:31.090035 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:31.090012    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pr9b6" podStartSLOduration=11.946901655 podStartE2EDuration="22.090006504s" totalImagesPullingTime="10.143104849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.169854178 +0000 UTC m=+23.361191872" watchObservedRunningTime="2026-09-17 17:41:31.090006504 +0000 UTC m=+24.281344201"
Sep 17 17:41:32.077899 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:32.077865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp" event={"ID":"a4885a76-c4ed-4eff-bf0e-424be1bb8a56","Type":"ContainerStarted","Data":"db5dc7bc847dab6bd31d6cc0dbe89a78d34ae8262ee3bda5f0cd89c7d0f29428"}
Sep 17 17:41:32.954511 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:32.954479    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:32.954511 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:32.954493    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:32.954722 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:32.954522    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:32.954722 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:32.954606    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:32.954722 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:32.954673    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:32.954906 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:32.954764    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:33.053409 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.053382    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:33.054030 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.054004    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:33.082911 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.082864    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhrbp" podStartSLOduration=4.344436155 podStartE2EDuration="24.08284713s" totalImagesPullingTime="19.738410975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:32.099953069 +0000 UTC m=+25.291290755" watchObservedRunningTime="2026-09-17 17:41:33.08284713 +0000 UTC m=+26.274184996"
Sep 17 17:41:33.084516 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.084498    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:41:33.084977 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.084954    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"26f2239a9047d45effceb652edfbbd74fb379ec7e2d3f3d6abdee8b20f331434"}
Sep 17 17:41:33.085343 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.085326    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:33.085786 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:33.085769    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pr9b6"
Sep 17 17:41:34.954751 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:34.954520    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:34.955198 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:34.954531    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:34.955198 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:34.954796    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:34.955198 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:34.954610    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:34.955198 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:34.954959    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:34.955198 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:34.955052    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:36.093121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.093083    2585 generic.go:356] "Generic (PLEG): container finished" podID="db93ed24-9b8e-49c4-96a4-164a1d955bb5" containerID="dff1e4086aa42f2539e7bb48af7fe468df01dd5648b368fe78a51b564bc53804" exitCode=0
Sep 17 17:41:36.093905 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.093165    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerDied","Data":"dff1e4086aa42f2539e7bb48af7fe468df01dd5648b368fe78a51b564bc53804"}
Sep 17 17:41:36.096834 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.096801    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:41:36.097167 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.097147    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"e5699a97d5a1e7dc61ee5b8905e1cffd96a85730ea7c9a10f9ddd01a0fae91a0"}
Sep 17 17:41:36.097430 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.097410    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:36.097535 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.097475    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:36.097535 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.097522    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:36.097608 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.097583    2585 scope.go:117] "RemoveContainer" containerID="8e641bf3f825dd7fb2ff0514cd960eb92e2bcfab2c3e865eab8d9050dde82c30"
Sep 17 17:41:36.113246 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.113226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:36.113856 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.113841    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:41:36.954382 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.954188    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:36.954581 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.954187    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:36.954581 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:36.954454    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:36.954581 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:36.954559    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:36.954721 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.954187    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:36.954721 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:36.954691    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:36.998670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:36.998632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kt64l"]
Sep 17 17:41:37.002547 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.002522    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29686"]
Sep 17 17:41:37.003150 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.003117    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6g6jp"]
Sep 17 17:41:37.104764 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.104722    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:41:37.105241 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.105221    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:37.105368 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:37.105345    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:37.106481 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.106461    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" event={"ID":"3075df7b-db2c-4495-8264-1a9102b2fe4b","Type":"ContainerStarted","Data":"87cda4dedbd31a196ffb2985b89cf5d003bed685b7cee8395911c78a9796baf1"}
Sep 17 17:41:37.106546 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.106538    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:37.106608 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:37.106594    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:37.106646 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.106627    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:37.106687 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:37.106676    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:37.137451 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:37.137411    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s" podStartSLOduration=10.424219987 podStartE2EDuration="28.137397194s" totalImagesPullingTime="17.713177207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:37.137178687 +0000 UTC m=+30.328516372" watchObservedRunningTime="2026-09-17 17:41:37.137397194 +0000 UTC m=+30.328734879"
Sep 17 17:41:38.109257 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:38.109228    2585 generic.go:356] "Generic (PLEG): container finished" podID="db93ed24-9b8e-49c4-96a4-164a1d955bb5" containerID="9e58f776166dfb5f7a2e3ff8a3c15ab19e5a98b19caa1fee99f3096dcc531b39" exitCode=0
Sep 17 17:41:38.109656 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:38.109305    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerDied","Data":"9e58f776166dfb5f7a2e3ff8a3c15ab19e5a98b19caa1fee99f3096dcc531b39"}
Sep 17 17:41:38.955039 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:38.955012    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:38.955188 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:38.955043    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:38.955188 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:38.955130    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:38.955277 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:38.955186    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:38.955277 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:38.955204    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:38.955277 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:38.955265    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:40.115950 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:40.115920    2585 generic.go:356] "Generic (PLEG): container finished" podID="db93ed24-9b8e-49c4-96a4-164a1d955bb5" containerID="a1213ea662ee6d0a31c028a4b816c42b00b11a7f7a76a53f0bd7a4cd03fde05f" exitCode=0
Sep 17 17:41:40.116326 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:40.115972    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerDied","Data":"a1213ea662ee6d0a31c028a4b816c42b00b11a7f7a76a53f0bd7a4cd03fde05f"}
Sep 17 17:41:40.954526 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:40.954492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:40.954700 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:40.954632    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29686" podUID="a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5"
Sep 17 17:41:40.954700 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:40.954492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:40.954842 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:40.954720    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kt64l" podUID="066cdf30-b263-4044-970e-46a388a81e48"
Sep 17 17:41:40.954842 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:40.954492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:40.954842 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:40.954794    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6jp" podUID="fa7b3894-1a3f-441c-80a0-7ff6ae96b749"
Sep 17 17:41:42.617607 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.617394    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:42.618011 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.617542    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:42.618011 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.617723    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret podName:fa7b3894-1a3f-441c-80a0-7ff6ae96b749 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:14.617702052 +0000 UTC m=+67.809039714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret") pod "global-pull-secret-syncer-6g6jp" (UID: "fa7b3894-1a3f-441c-80a0-7ff6ae96b749") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:42.688552 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.688475    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeReady"
Sep 17 17:41:42.688742 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.688595    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:41:42.717968 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.717938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:42.718124 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.718101    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:42.718188 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.718160    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:14.718142623 +0000 UTC m=+67.909480300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:42.738898 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.738873    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t2nm9"]
Sep 17 17:41:42.758509 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.757748    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gjxg6"]
Sep 17 17:41:42.758509 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.757879    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:42.760620 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.760598    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:41:42.760620 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.760619    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:41:42.760937 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.760920    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4db82\""
Sep 17 17:41:42.773894 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.773872    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2nm9"]
Sep 17 17:41:42.774008 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.773918    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjxg6"]
Sep 17 17:41:42.774008 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.773960    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:42.777152 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.777134    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:41:42.777152 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.777149    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2krvt\""
Sep 17 17:41:42.777304 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.777139    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:41:42.777304 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.777228    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:41:42.818749 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.818723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:42.818909 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.818894    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:42.818977 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.818917    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:42.818977 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.818928    2585 projected.go:196] Error preparing data for projected volume kube-api-access-5fqs6 for pod openshift-network-diagnostics/network-check-target-kt64l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:42.819070 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:42.818983    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6 podName:066cdf30-b263-4044-970e-46a388a81e48 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:14.818967308 +0000 UTC m=+68.010304973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fqs6" (UniqueName: "kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6") pod "network-check-target-kt64l" (UID: "066cdf30-b263-4044-970e-46a388a81e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:42.919885 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.919855    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5792e032-4987-4024-984f-950180feebc9-tmp-dir\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:42.920069 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.919897    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k896l\" (UniqueName: \"kubernetes.io/projected/800607ce-206f-4ce9-8edc-d6636d819654-kube-api-access-k896l\") pod \"ingress-canary-gjxg6\" (UID: \"800607ce-206f-4ce9-8edc-d6636d819654\") " pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:42.920069 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.919963    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cp2m\" (UniqueName: \"kubernetes.io/projected/5792e032-4987-4024-984f-950180feebc9-kube-api-access-8cp2m\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:42.920069 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.920037    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:42.920069 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.920056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/800607ce-206f-4ce9-8edc-d6636d819654-cert\") pod \"ingress-canary-gjxg6\" (UID: \"800607ce-206f-4ce9-8edc-d6636d819654\") " pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:42.920244 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.920097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5792e032-4987-4024-984f-950180feebc9-config-volume\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:42.953860 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.953784    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:41:42.953988 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.953790    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:41:42.953988 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.953905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:41:42.958560 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.958233    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8q2r\""
Sep 17 17:41:42.958560 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.958311    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nqf6f\""
Sep 17 17:41:42.958862 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.958846    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:42.959098 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.959079    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:42.959188 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.959112    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:42.959616 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:42.959602    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:43.021005 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.020980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.021112 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/800607ce-206f-4ce9-8edc-d6636d819654-cert\") pod \"ingress-canary-gjxg6\" (UID: \"800607ce-206f-4ce9-8edc-d6636d819654\") " pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:43.021112 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5792e032-4987-4024-984f-950180feebc9-config-volume\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.021112 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021109    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5792e032-4987-4024-984f-950180feebc9-tmp-dir\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.021281 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:43.021132    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:43.021281 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k896l\" (UniqueName: \"kubernetes.io/projected/800607ce-206f-4ce9-8edc-d6636d819654-kube-api-access-k896l\") pod \"ingress-canary-gjxg6\" (UID: \"800607ce-206f-4ce9-8edc-d6636d819654\") " pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:43.021281 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:43.021197    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls podName:5792e032-4987-4024-984f-950180feebc9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:43.521175388 +0000 UTC m=+36.712513053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls") pod "dns-default-t2nm9" (UID: "5792e032-4987-4024-984f-950180feebc9") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:43.021281 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021251    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cp2m\" (UniqueName: \"kubernetes.io/projected/5792e032-4987-4024-984f-950180feebc9-kube-api-access-8cp2m\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.021482 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021455    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5792e032-4987-4024-984f-950180feebc9-tmp-dir\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.021701 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.021684    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5792e032-4987-4024-984f-950180feebc9-config-volume\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.025797 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.025778    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/800607ce-206f-4ce9-8edc-d6636d819654-cert\") pod \"ingress-canary-gjxg6\" (UID: \"800607ce-206f-4ce9-8edc-d6636d819654\") " pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:43.037721 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.037700    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cp2m\" (UniqueName: \"kubernetes.io/projected/5792e032-4987-4024-984f-950180feebc9-kube-api-access-8cp2m\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.041753 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.041732    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k896l\" (UniqueName: \"kubernetes.io/projected/800607ce-206f-4ce9-8edc-d6636d819654-kube-api-access-k896l\") pod \"ingress-canary-gjxg6\" (UID: \"800607ce-206f-4ce9-8edc-d6636d819654\") " pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:43.084328 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.084303    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjxg6"
Sep 17 17:41:43.525436 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:43.525402    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:43.525658 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:43.525575    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:43.525658 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:43.525655    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls podName:5792e032-4987-4024-984f-950180feebc9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:44.525632403 +0000 UTC m=+37.716970074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls") pod "dns-default-t2nm9" (UID: "5792e032-4987-4024-984f-950180feebc9") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:44.444434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:44.444407    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjxg6"]
Sep 17 17:41:44.447517 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:41:44.447494    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800607ce_206f_4ce9_8edc_d6636d819654.slice/crio-4f73559d40c7a7cfb39c59b6a28e417b61c93608f560048facdb4caa9c3ddeca WatchSource:0}: Error finding container 4f73559d40c7a7cfb39c59b6a28e417b61c93608f560048facdb4caa9c3ddeca: Status 404 returned error can't find the container with id 4f73559d40c7a7cfb39c59b6a28e417b61c93608f560048facdb4caa9c3ddeca
Sep 17 17:41:44.533240 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:44.533214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:44.533391 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:44.533374    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:44.533443 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:44.533433    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls podName:5792e032-4987-4024-984f-950180feebc9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:46.533419052 +0000 UTC m=+39.724756715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls") pod "dns-default-t2nm9" (UID: "5792e032-4987-4024-984f-950180feebc9") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:45.132105 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:45.132073    2585 generic.go:356] "Generic (PLEG): container finished" podID="db93ed24-9b8e-49c4-96a4-164a1d955bb5" containerID="703e1a5e71968147c76ef78ec5476b04458d615304d03dab7357afb9528c131a" exitCode=0
Sep 17 17:41:45.132260 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:45.132149    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerDied","Data":"703e1a5e71968147c76ef78ec5476b04458d615304d03dab7357afb9528c131a"}
Sep 17 17:41:45.133645 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:45.133617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjxg6" event={"ID":"800607ce-206f-4ce9-8edc-d6636d819654","Type":"ContainerStarted","Data":"4f73559d40c7a7cfb39c59b6a28e417b61c93608f560048facdb4caa9c3ddeca"}
Sep 17 17:41:46.140249 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:46.140054    2585 generic.go:356] "Generic (PLEG): container finished" podID="db93ed24-9b8e-49c4-96a4-164a1d955bb5" containerID="6a9a987731dbd2db2208e5987a612036a655a64ac0da516e8a0e6fa61016f744" exitCode=0
Sep 17 17:41:46.140741 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:46.140106    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerDied","Data":"6a9a987731dbd2db2208e5987a612036a655a64ac0da516e8a0e6fa61016f744"}
Sep 17 17:41:46.548415 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:46.548394    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:46.548518 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:46.548507    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:46.548563 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:46.548555    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls podName:5792e032-4987-4024-984f-950180feebc9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:50.548542195 +0000 UTC m=+43.739879858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls") pod "dns-default-t2nm9" (UID: "5792e032-4987-4024-984f-950180feebc9") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:47.146247 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:47.146221    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" event={"ID":"db93ed24-9b8e-49c4-96a4-164a1d955bb5","Type":"ContainerStarted","Data":"9f92a962af6b9473fe2baa0fb87d860e5d9fe5f062cd0bba39de4379621b9316"}
Sep 17 17:41:47.147691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:47.147669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjxg6" event={"ID":"800607ce-206f-4ce9-8edc-d6636d819654","Type":"ContainerStarted","Data":"2c5c741736c482155c4885ce33110e4c669171eacca63a763432cf115de1092e"}
Sep 17 17:41:47.183432 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:47.183388    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sbbqb" podStartSLOduration=10.101290517 podStartE2EDuration="38.183374513s" totalImagesPullingTime="28.082083996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:47.182956241 +0000 UTC m=+40.374293923" watchObservedRunningTime="2026-09-17 17:41:47.183374513 +0000 UTC m=+40.374712197"
Sep 17 17:41:50.572015 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:50.571977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:50.572425 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:50.572118    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:50.572425 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:50.572175    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls podName:5792e032-4987-4024-984f-950180feebc9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:58.572162311 +0000 UTC m=+51.763499973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls") pod "dns-default-t2nm9" (UID: "5792e032-4987-4024-984f-950180feebc9") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:58.626143 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:41:58.626108    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:41:58.626511 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:58.626264    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:58.626511 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:41:58.626339    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls podName:5792e032-4987-4024-984f-950180feebc9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:14.626323943 +0000 UTC m=+67.817661606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls") pod "dns-default-t2nm9" (UID: "5792e032-4987-4024-984f-950180feebc9") : secret "dns-default-metrics-tls" not found
Sep 17 17:42:00.455365 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.455307    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gjxg6" podStartSLOduration=16.371106251 podStartE2EDuration="18.455290426s" totalImagesPullingTime="2.084184175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:47.214873721 +0000 UTC m=+40.406211405" watchObservedRunningTime="2026-09-17 17:42:00.455290426 +0000 UTC m=+53.646628128"
Sep 17 17:42:00.455934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.455611    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"]
Sep 17 17:42:00.481760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.481737    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"]
Sep 17 17:42:00.481911 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.481865    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"
Sep 17 17:42:00.484901 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.484869    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xz9kv\""
Sep 17 17:42:00.485125 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.485109    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:42:00.485211 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.485140    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:42:00.493724 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.493706    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"]
Sep 17 17:42:00.493803 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.493734    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"]
Sep 17 17:42:00.493860 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.493801    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.496908 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.496889    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:42:00.497256 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.497235    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:42:00.497355 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.497265    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:42:00.497675 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.497658    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w44j6\""
Sep 17 17:42:00.497758 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.497658    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:42:00.538482 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.538458    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29df95df-3880-4cf6-9de8-caa826016278-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.538580 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.538499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqn4x\" (UniqueName: \"kubernetes.io/projected/29df95df-3880-4cf6-9de8-caa826016278-kube-api-access-pqn4x\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.538580 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.538562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmr6j\" (UniqueName: \"kubernetes.io/projected/202532de-c9e0-4705-9e6c-3a8252b95e44-kube-api-access-lmr6j\") pod \"volume-data-source-validator-5c4db8bdf9-pzxc8\" (UID: \"202532de-c9e0-4705-9e6c-3a8252b95e44\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"
Sep 17 17:42:00.538648 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.538589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29df95df-3880-4cf6-9de8-caa826016278-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.560163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.560141    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"]
Sep 17 17:42:00.585270 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.585251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:00.602862 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.602837    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:42:00.602959 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.602865    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:42:00.603163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.603147    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rzwm7\""
Sep 17 17:42:00.603261 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.603191    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:42:00.604176 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.604158    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"]
Sep 17 17:42:00.639206 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639189    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29df95df-3880-4cf6-9de8-caa826016278-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.639291 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29df95df-3880-4cf6-9de8-caa826016278-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.639291 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639247    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcsl\" (UniqueName: \"kubernetes.io/projected/de15382c-054f-42bf-85ac-abe223487b32-kube-api-access-wzcsl\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:00.639401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639378    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqn4x\" (UniqueName: \"kubernetes.io/projected/29df95df-3880-4cf6-9de8-caa826016278-kube-api-access-pqn4x\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.639451 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:00.639485 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmr6j\" (UniqueName: \"kubernetes.io/projected/202532de-c9e0-4705-9e6c-3a8252b95e44-kube-api-access-lmr6j\") pod \"volume-data-source-validator-5c4db8bdf9-pzxc8\" (UID: \"202532de-c9e0-4705-9e6c-3a8252b95e44\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"
Sep 17 17:42:00.639835 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.639798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29df95df-3880-4cf6-9de8-caa826016278-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.642419 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.642401    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29df95df-3880-4cf6-9de8-caa826016278-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.667109 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.667083    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqn4x\" (UniqueName: \"kubernetes.io/projected/29df95df-3880-4cf6-9de8-caa826016278-kube-api-access-pqn4x\") pod \"kube-storage-version-migrator-operator-55889f7b47-892xx\" (UID: \"29df95df-3880-4cf6-9de8-caa826016278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.670651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.670635    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmr6j\" (UniqueName: \"kubernetes.io/projected/202532de-c9e0-4705-9e6c-3a8252b95e44-kube-api-access-lmr6j\") pod \"volume-data-source-validator-5c4db8bdf9-pzxc8\" (UID: \"202532de-c9e0-4705-9e6c-3a8252b95e44\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"
Sep 17 17:42:00.740748 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.740692    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:00.740748 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.740742    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcsl\" (UniqueName: \"kubernetes.io/projected/de15382c-054f-42bf-85ac-abe223487b32-kube-api-access-wzcsl\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:00.740899 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:00.740845    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:42:00.740899 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:00.740898    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls podName:de15382c-054f-42bf-85ac-abe223487b32 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:01.24088577 +0000 UTC m=+54.432223432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mkzgc" (UID: "de15382c-054f-42bf-85ac-abe223487b32") : secret "samples-operator-tls" not found
Sep 17 17:42:00.765741 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.765713    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcsl\" (UniqueName: \"kubernetes.io/projected/de15382c-054f-42bf-85ac-abe223487b32-kube-api-access-wzcsl\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:00.786135 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.786110    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ff67d7dcd-htsxj"]
Sep 17 17:42:00.790399 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.790382    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"
Sep 17 17:42:00.801549 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.801530    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"
Sep 17 17:42:00.809020 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.809007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.815645 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.815628    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:42:00.816857 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.816837    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:42:00.817224 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.817205    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lvrnq\""
Sep 17 17:42:00.818684 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.818655    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:42:00.825438 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.825392    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:42:00.838478 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.838456    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff67d7dcd-htsxj"]
Sep 17 17:42:00.841530 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841510    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-image-registry-private-configuration\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841629 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841537    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-certificates\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841629 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841580    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v95r\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-kube-api-access-8v95r\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841629 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841600    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dc6558d-93df-4e43-99dd-a49e53ad28df-ca-trust-extracted\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-installation-pull-secrets\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841992 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-trusted-ca\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.841992 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.841802    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-bound-sa-token\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.942965 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.942935    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v95r\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-kube-api-access-8v95r\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.942965 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.942966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943126 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.942983    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dc6558d-93df-4e43-99dd-a49e53ad28df-ca-trust-extracted\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943126 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.942998    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-installation-pull-secrets\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943126 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.943019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-trusted-ca\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943126 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.943037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-bound-sa-token\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943322 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:00.943296    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:42:00.943381 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:00.943321    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff67d7dcd-htsxj: secret "image-registry-tls" not found
Sep 17 17:42:00.943381 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.943355    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-image-registry-private-configuration\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943484 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:00.943393    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls podName:4dc6558d-93df-4e43-99dd-a49e53ad28df nodeName:}" failed. No retries permitted until 2026-09-17 17:42:01.443371398 +0000 UTC m=+54.634709076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls") pod "image-registry-ff67d7dcd-htsxj" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df") : secret "image-registry-tls" not found
Sep 17 17:42:00.943484 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.943406    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dc6558d-93df-4e43-99dd-a49e53ad28df-ca-trust-extracted\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.943484 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.943428    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-certificates\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.944169 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.944055    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-certificates\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.944262 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.944189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-trusted-ca\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.945605 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.945582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-installation-pull-secrets\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.945605 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.945601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-image-registry-private-configuration\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.954380 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.954363    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v95r\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-kube-api-access-8v95r\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:00.954461 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:00.954445    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-bound-sa-token\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:01.083592 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:01.083456    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8"]
Sep 17 17:42:01.085908 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:01.085882    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202532de_c9e0_4705_9e6c_3a8252b95e44.slice/crio-910ea9667cd104e3f3c1c4b7ab435296bbf3df3ccbe72fe3dd6fc09cafc1366c WatchSource:0}: Error finding container 910ea9667cd104e3f3c1c4b7ab435296bbf3df3ccbe72fe3dd6fc09cafc1366c: Status 404 returned error can't find the container with id 910ea9667cd104e3f3c1c4b7ab435296bbf3df3ccbe72fe3dd6fc09cafc1366c
Sep 17 17:42:01.129435 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:01.129413    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx"]
Sep 17 17:42:01.130078 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:01.130059    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29df95df_3880_4cf6_9de8_caa826016278.slice/crio-8f886ea66c29e440ceb1c62b141f4509e30f1aab0927f6b480d043fb69b8cf27 WatchSource:0}: Error finding container 8f886ea66c29e440ceb1c62b141f4509e30f1aab0927f6b480d043fb69b8cf27: Status 404 returned error can't find the container with id 8f886ea66c29e440ceb1c62b141f4509e30f1aab0927f6b480d043fb69b8cf27
Sep 17 17:42:01.181198 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:01.181165    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx" event={"ID":"29df95df-3880-4cf6-9de8-caa826016278","Type":"ContainerStarted","Data":"8f886ea66c29e440ceb1c62b141f4509e30f1aab0927f6b480d043fb69b8cf27"}
Sep 17 17:42:01.182306 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:01.182287    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8" event={"ID":"202532de-c9e0-4705-9e6c-3a8252b95e44","Type":"ContainerStarted","Data":"910ea9667cd104e3f3c1c4b7ab435296bbf3df3ccbe72fe3dd6fc09cafc1366c"}
Sep 17 17:42:01.244895 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:01.244868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:01.244986 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:01.244973    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:42:01.245033 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:01.245025    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls podName:de15382c-054f-42bf-85ac-abe223487b32 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:02.245007573 +0000 UTC m=+55.436345253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mkzgc" (UID: "de15382c-054f-42bf-85ac-abe223487b32") : secret "samples-operator-tls" not found
Sep 17 17:42:01.446969 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:01.446943    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:01.447079 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:01.447042    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:42:01.447079 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:01.447052    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff67d7dcd-htsxj: secret "image-registry-tls" not found
Sep 17 17:42:01.447149 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:01.447095    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls podName:4dc6558d-93df-4e43-99dd-a49e53ad28df nodeName:}" failed. No retries permitted until 2026-09-17 17:42:02.447083291 +0000 UTC m=+55.638420953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls") pod "image-registry-ff67d7dcd-htsxj" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df") : secret "image-registry-tls" not found
Sep 17 17:42:02.253321 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:02.253284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:02.253774 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:02.253426    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:42:02.253774 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:02.253481    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls podName:de15382c-054f-42bf-85ac-abe223487b32 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.253463315 +0000 UTC m=+57.444800995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mkzgc" (UID: "de15382c-054f-42bf-85ac-abe223487b32") : secret "samples-operator-tls" not found
Sep 17 17:42:02.454440 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:02.454396    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:02.454618 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:02.454550    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:42:02.454618 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:02.454567    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff67d7dcd-htsxj: secret "image-registry-tls" not found
Sep 17 17:42:02.454730 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:02.454627    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls podName:4dc6558d-93df-4e43-99dd-a49e53ad28df nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.454607825 +0000 UTC m=+57.645945509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls") pod "image-registry-ff67d7dcd-htsxj" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df") : secret "image-registry-tls" not found
Sep 17 17:42:04.191712 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.191673    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx" event={"ID":"29df95df-3880-4cf6-9de8-caa826016278","Type":"ContainerStarted","Data":"eda1457801ae8e798da6e254a8f173ebfc2ad1602548d743e4293941782aa7cf"}
Sep 17 17:42:04.193176 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.193148    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8" event={"ID":"202532de-c9e0-4705-9e6c-3a8252b95e44","Type":"ContainerStarted","Data":"eb1c44e5583c03c3425bb149f6e7bec98a903af0786073d8dd242fc08a0354ff"}
Sep 17 17:42:04.212632 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.212591    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx" podStartSLOduration=1.759192778 podStartE2EDuration="4.212578016s" totalImagesPullingTime="2.453385238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:04.211200751 +0000 UTC m=+57.402538437" watchObservedRunningTime="2026-09-17 17:42:04.212578016 +0000 UTC m=+57.403915701"
Sep 17 17:42:04.238513 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.238469    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pzxc8" podStartSLOduration=1.746090361 podStartE2EDuration="4.238456096s" totalImagesPullingTime="2.492365735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:04.2381294 +0000 UTC m=+57.429467088" watchObservedRunningTime="2026-09-17 17:42:04.238456096 +0000 UTC m=+57.429793780"
Sep 17 17:42:04.268602 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.268561    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:04.268801 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:04.268778    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:42:04.268909 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:04.268862    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls podName:de15382c-054f-42bf-85ac-abe223487b32 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.268845322 +0000 UTC m=+61.460182985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mkzgc" (UID: "de15382c-054f-42bf-85ac-abe223487b32") : secret "samples-operator-tls" not found
Sep 17 17:42:04.469637 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.469559    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:04.469754 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:04.469689    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:42:04.469754 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:04.469702    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff67d7dcd-htsxj: secret "image-registry-tls" not found
Sep 17 17:42:04.469840 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:04.469760    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls podName:4dc6558d-93df-4e43-99dd-a49e53ad28df nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.469742091 +0000 UTC m=+61.661079760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls") pod "image-registry-ff67d7dcd-htsxj" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df") : secret "image-registry-tls" not found
Sep 17 17:42:04.879398 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.879370    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zbcjh"]
Sep 17 17:42:04.913691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.913669    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:04.917429 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.917393    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:42:04.917560 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.917537    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cb5pj\""
Sep 17 17:42:04.927603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.927583    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:42:04.927603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.927595    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:42:04.927734 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.927586    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:42:04.929658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.929638    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbcjh"]
Sep 17 17:42:04.972475 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.972454    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-crio-socket\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:04.972566 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.972487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-data-volume\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:04.972566 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.972508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:04.972566 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.972527    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:04.972691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:04.972643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nh7\" (UniqueName: \"kubernetes.io/projected/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-api-access-v6nh7\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073109 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073080    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-crio-socket\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073210 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-data-volume\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073210 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073147    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073210 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073374 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073206    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-crio-socket\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073374 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nh7\" (UniqueName: \"kubernetes.io/projected/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-api-access-v6nh7\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.073374 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.073280    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:05.073374 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.073320    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:05.073374 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.073341    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:05.573324274 +0000 UTC m=+58.764661954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:05.073374 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.073374    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:05.573360517 +0000 UTC m=+58.764698180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:05.073577 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.073417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-data-volume\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.085643 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.085616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nh7\" (UniqueName: \"kubernetes.io/projected/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-api-access-v6nh7\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.576917 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.576883    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.576917 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:05.576921    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:05.577337 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.577005    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:05.577337 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.577029    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:05.577337 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.577057    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.57704381 +0000 UTC m=+59.768381472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:05.577337 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:05.577089    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.577076362 +0000 UTC m=+59.768414025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:06.585497 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.585463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:06.585497 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.585499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:06.585992 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:06.585572    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:06.585992 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:06.585602    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:06.585992 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:06.585629    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.585615951 +0000 UTC m=+61.776953617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:06.585992 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:06.585652    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.585639212 +0000 UTC m=+61.776976875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:06.592015 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.591999    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nvkjm_9e6f7109-7114-4a3e-922f-408eb432d5dd/dns-node-resolver/0.log"
Sep 17 17:42:06.874035 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.874008    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-s5sfz"]
Sep 17 17:42:06.894573 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.894548    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-s5sfz"]
Sep 17 17:42:06.894692 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.894634    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:06.897537 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.897512    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m6bcw\""
Sep 17 17:42:06.897662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.897539    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:42:06.897662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.897517    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:42:06.898591 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.898569    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:42:06.898693 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.898591    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:42:06.989171 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.989147    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-signing-key\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:06.989290 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.989210    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-signing-cabundle\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:06.989290 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:06.989242    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcld\" (UniqueName: \"kubernetes.io/projected/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-kube-api-access-pjcld\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.089627 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.089590    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-signing-cabundle\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.089627 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.089628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcld\" (UniqueName: \"kubernetes.io/projected/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-kube-api-access-pjcld\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.089881 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.089825    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-signing-key\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.090231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.090212    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-signing-cabundle\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.092237 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.092211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-signing-key\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.099773 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.099752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcld\" (UniqueName: \"kubernetes.io/projected/0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07-kube-api-access-pjcld\") pod \"service-ca-55f6fc7b86-s5sfz\" (UID: \"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07\") " pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.203795 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.203741    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz"
Sep 17 17:42:07.392930 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.392906    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kggnz_2177cca8-a58d-4b81-8a10-65f67ee8ada0/node-ca/0.log"
Sep 17 17:42:07.560662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:07.560597    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-s5sfz"]
Sep 17 17:42:07.564226 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:07.564198    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2d4d43_eba0_4d9c_abfb_39cb7a4f2a07.slice/crio-8c3c8f91cf96a183b639a46eeda70b4663b1099496194174fa3291041bee281e WatchSource:0}: Error finding container 8c3c8f91cf96a183b639a46eeda70b4663b1099496194174fa3291041bee281e: Status 404 returned error can't find the container with id 8c3c8f91cf96a183b639a46eeda70b4663b1099496194174fa3291041bee281e
Sep 17 17:42:08.130627 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.130596    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9d4s"
Sep 17 17:42:08.204854 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.204744    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz" event={"ID":"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07","Type":"ContainerStarted","Data":"8c3c8f91cf96a183b639a46eeda70b4663b1099496194174fa3291041bee281e"}
Sep 17 17:42:08.299631 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.299594    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:08.299793 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.299757    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:42:08.299896 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.299847    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls podName:de15382c-054f-42bf-85ac-abe223487b32 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:16.299827207 +0000 UTC m=+69.491164888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mkzgc" (UID: "de15382c-054f-42bf-85ac-abe223487b32") : secret "samples-operator-tls" not found
Sep 17 17:42:08.501376 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.501302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:08.501505 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.501453    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:42:08.501505 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.501473    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff67d7dcd-htsxj: secret "image-registry-tls" not found
Sep 17 17:42:08.501571 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.501527    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls podName:4dc6558d-93df-4e43-99dd-a49e53ad28df nodeName:}" failed. No retries permitted until 2026-09-17 17:42:16.501512637 +0000 UTC m=+69.692850300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls") pod "image-registry-ff67d7dcd-htsxj" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df") : secret "image-registry-tls" not found
Sep 17 17:42:08.593217 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.593179    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjxg6_800607ce-206f-4ce9-8edc-d6636d819654/serve-healthcheck-canary/0.log"
Sep 17 17:42:08.602352 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.602325    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:08.602453 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:08.602358    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:08.602491 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.602473    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:08.602491 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.602473    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:08.602550 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.602536    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:12.602520047 +0000 UTC m=+65.793857710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:08.602550 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:08.602549    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:12.60254323 +0000 UTC m=+65.793880893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:10.211324 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:10.211292    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz" event={"ID":"0f2d4d43-eba0-4d9c-abfb-39cb7a4f2a07","Type":"ContainerStarted","Data":"e60c6951a93a2b3fea5ba7b7691b9c7a236a77bde8138abab7450edca1956a6c"}
Sep 17 17:42:10.229965 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:10.229914    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-s5sfz" podStartSLOduration=2.382901233 podStartE2EDuration="4.229894206s" totalImagesPullingTime="1.846992973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:10.229498076 +0000 UTC m=+63.420835763" watchObservedRunningTime="2026-09-17 17:42:10.229894206 +0000 UTC m=+63.421231892"
Sep 17 17:42:12.635691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:12.635655    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:12.636072 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:12.635698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:12.636072 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:12.635787    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:12.636072 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:12.635829    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:12.636072 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:12.635872    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:20.635853164 +0000 UTC m=+73.827190844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:12.636072 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:12.635888    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:20.635879871 +0000 UTC m=+73.827217537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:14.651826 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.651781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:42:14.652173 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.651881    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:42:14.654212 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.654195    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5792e032-4987-4024-984f-950180feebc9-metrics-tls\") pod \"dns-default-t2nm9\" (UID: \"5792e032-4987-4024-984f-950180feebc9\") " pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:42:14.654641 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.654626    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:42:14.664483 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.664456    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa7b3894-1a3f-441c-80a0-7ff6ae96b749-original-pull-secret\") pod \"global-pull-secret-syncer-6g6jp\" (UID: \"fa7b3894-1a3f-441c-80a0-7ff6ae96b749\") " pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:42:14.752417 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.752395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:42:14.755151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.755136    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:42:14.763552 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:14.763535    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:42:14.763624 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:14.763582    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs podName:a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:43:18.763568584 +0000 UTC m=+131.954906247 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs") pod "network-metrics-daemon-29686" (UID: "a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5") : secret "metrics-daemon-secret" not found
Sep 17 17:42:14.771421 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.771406    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6jp"
Sep 17 17:42:14.853602 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.853494    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:42:14.857942 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.857922    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:42:14.866977 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.866959    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:42:14.871945 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.871932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4db82\""
Sep 17 17:42:14.877389 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.877361    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqs6\" (UniqueName: \"kubernetes.io/projected/066cdf30-b263-4044-970e-46a388a81e48-kube-api-access-5fqs6\") pod \"network-check-target-kt64l\" (UID: \"066cdf30-b263-4044-970e-46a388a81e48\") " pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:42:14.880153 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:14.880133    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:42:15.054702 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.054677    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6g6jp"]
Sep 17 17:42:15.057245 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:15.057223    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7b3894_1a3f_441c_80a0_7ff6ae96b749.slice/crio-bc68f352256afa19329a75f13e83cee83bd92e997f558db9b13b4de6a2e7406a WatchSource:0}: Error finding container bc68f352256afa19329a75f13e83cee83bd92e997f558db9b13b4de6a2e7406a: Status 404 returned error can't find the container with id bc68f352256afa19329a75f13e83cee83bd92e997f558db9b13b4de6a2e7406a
Sep 17 17:42:15.067598 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.067578    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8q2r\""
Sep 17 17:42:15.075781 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.075764    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:42:15.183542 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.183521    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2nm9"]
Sep 17 17:42:15.185167 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:15.185142    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5792e032_4987_4024_984f_950180feebc9.slice/crio-22cb4a59d2a7e0312aef1d72054bf71fdd3318fd6588ec38f16620b15f34d1b7 WatchSource:0}: Error finding container 22cb4a59d2a7e0312aef1d72054bf71fdd3318fd6588ec38f16620b15f34d1b7: Status 404 returned error can't find the container with id 22cb4a59d2a7e0312aef1d72054bf71fdd3318fd6588ec38f16620b15f34d1b7
Sep 17 17:42:15.226108 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.226076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2nm9" event={"ID":"5792e032-4987-4024-984f-950180feebc9","Type":"ContainerStarted","Data":"22cb4a59d2a7e0312aef1d72054bf71fdd3318fd6588ec38f16620b15f34d1b7"}
Sep 17 17:42:15.227194 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.227172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6g6jp" event={"ID":"fa7b3894-1a3f-441c-80a0-7ff6ae96b749","Type":"ContainerStarted","Data":"bc68f352256afa19329a75f13e83cee83bd92e997f558db9b13b4de6a2e7406a"}
Sep 17 17:42:15.350360 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:15.350333    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kt64l"]
Sep 17 17:42:15.354669 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:15.354649    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066cdf30_b263_4044_970e_46a388a81e48.slice/crio-9fe5554cb1d3dfb2c89876fad26972c944998bb1dbf5bead21adb64c9771304f WatchSource:0}: Error finding container 9fe5554cb1d3dfb2c89876fad26972c944998bb1dbf5bead21adb64c9771304f: Status 404 returned error can't find the container with id 9fe5554cb1d3dfb2c89876fad26972c944998bb1dbf5bead21adb64c9771304f
Sep 17 17:42:16.232719 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.232681    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kt64l" event={"ID":"066cdf30-b263-4044-970e-46a388a81e48","Type":"ContainerStarted","Data":"9fe5554cb1d3dfb2c89876fad26972c944998bb1dbf5bead21adb64c9771304f"}
Sep 17 17:42:16.365664 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.365619    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:16.368901 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.368857    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de15382c-054f-42bf-85ac-abe223487b32-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mkzgc\" (UID: \"de15382c-054f-42bf-85ac-abe223487b32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:16.496607 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.496541    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rzwm7\""
Sep 17 17:42:16.504308 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.504071    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"
Sep 17 17:42:16.568004 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.567522    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:16.571214 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.571168    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"image-registry-ff67d7dcd-htsxj\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") " pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:16.727709 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.727508    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lvrnq\""
Sep 17 17:42:16.735355 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:16.735095    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:17.422359 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:17.422285    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc"]
Sep 17 17:42:17.428025 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:17.427999    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff67d7dcd-htsxj"]
Sep 17 17:42:18.242308 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.241797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2nm9" event={"ID":"5792e032-4987-4024-984f-950180feebc9","Type":"ContainerStarted","Data":"3888fff6f52aa98d15c1df3782cc56159fc6ed218b86389e6af0b59765cf710f"}
Sep 17 17:42:18.242308 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.241882    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2nm9" event={"ID":"5792e032-4987-4024-984f-950180feebc9","Type":"ContainerStarted","Data":"1c3f3f5769cae23d20828f47b205b4d9979f72ea324e1dc467cd51694660ea75"}
Sep 17 17:42:18.242308 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.241934    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:42:18.243250 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.243220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc" event={"ID":"de15382c-054f-42bf-85ac-abe223487b32","Type":"ContainerStarted","Data":"31d04d43bdfed3010d36a98278af2553768fa22fab0285e3fd7248d4a26f7a7b"}
Sep 17 17:42:18.244866 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.244838    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj" event={"ID":"4dc6558d-93df-4e43-99dd-a49e53ad28df","Type":"ContainerStarted","Data":"03c865ae0a45589615ae587bfd1c293de91f783d81fbb26018792060cd17ab54"}
Sep 17 17:42:18.244997 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.244874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj" event={"ID":"4dc6558d-93df-4e43-99dd-a49e53ad28df","Type":"ContainerStarted","Data":"276dba731937a4637174e84d85b0628abebd1a54ae2ea18b6b7188b6041550ce"}
Sep 17 17:42:18.244997 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.244985    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:18.274385 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.274342    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t2nm9" podStartSLOduration=34.291409468 podStartE2EDuration="36.274328547s" totalImagesPullingTime="1.982919079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:18.269838653 +0000 UTC m=+71.461176340" watchObservedRunningTime="2026-09-17 17:42:18.274328547 +0000 UTC m=+71.465666232"
Sep 17 17:42:18.298309 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:18.298270    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj" podStartSLOduration=18.298249913 podStartE2EDuration="18.298249913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:18.29671629 +0000 UTC m=+71.488053997" watchObservedRunningTime="2026-09-17 17:42:18.298249913 +0000 UTC m=+71.489587616"
Sep 17 17:42:20.699788 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:20.699716    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:20.700206 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:20.699832    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:20.700206 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:20.699943    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:20.700206 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:20.699989    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls podName:1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:36.699976613 +0000 UTC m=+89.891314276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbcjh" (UID: "1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:20.700380 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:20.700356    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:21.257669 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.257594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc" event={"ID":"de15382c-054f-42bf-85ac-abe223487b32","Type":"ContainerStarted","Data":"bdec5189fc72aa2cb46232bffc34f93d432b622cd1482b8ec185a8d380faebab"}
Sep 17 17:42:21.257838 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.257672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc" event={"ID":"de15382c-054f-42bf-85ac-abe223487b32","Type":"ContainerStarted","Data":"d1107dcefe145a3fab7c7beb5a4bb7ef180bc1dd2623fad5b4d1442198840900"}
Sep 17 17:42:21.259110 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.259080    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6g6jp" event={"ID":"fa7b3894-1a3f-441c-80a0-7ff6ae96b749","Type":"ContainerStarted","Data":"0751f7efe827fa929247d5d9c677e3dc3f61bb1ef033163b92d2a3835a611588"}
Sep 17 17:42:21.260420 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.260400    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kt64l" event={"ID":"066cdf30-b263-4044-970e-46a388a81e48","Type":"ContainerStarted","Data":"64c1c946edeb499c7804ba46e524ad12bb72d518e9060825404f5a57cd284b49"}
Sep 17 17:42:21.260587 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.260571    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:42:21.287108 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.287074    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mkzgc" podStartSLOduration=17.893151917 podStartE2EDuration="21.28706096s" totalImagesPullingTime="3.393909043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:21.286863659 +0000 UTC m=+74.478201344" watchObservedRunningTime="2026-09-17 17:42:21.28706096 +0000 UTC m=+74.478398683"
Sep 17 17:42:21.312739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.312706    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6g6jp" podStartSLOduration=66.528856776 podStartE2EDuration="1m12.312693772s" totalImagesPullingTime="5.783836996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:21.309769656 +0000 UTC m=+74.501107341" watchObservedRunningTime="2026-09-17 17:42:21.312693772 +0000 UTC m=+74.504031457"
Sep 17 17:42:21.361967 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:21.361927    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kt64l" podStartSLOduration=66.887371835 podStartE2EDuration="1m12.361914113s" totalImagesPullingTime="5.474542278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:21.361067492 +0000 UTC m=+74.552405176" watchObservedRunningTime="2026-09-17 17:42:21.361914113 +0000 UTC m=+74.553251798"
Sep 17 17:42:28.251088 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:28.251055    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2nm9"
Sep 17 17:42:31.205747 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.205714    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"]
Sep 17 17:42:31.210221 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.210199    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.214686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.214661    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:42:31.214686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.214677    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8lddb\""
Sep 17 17:42:31.216302 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.216222    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:42:31.216302 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.216228    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:42:31.216410 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.216296    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:42:31.221116 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.221097    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff67d7dcd-htsxj"]
Sep 17 17:42:31.232417 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.232397    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"]
Sep 17 17:42:31.269922 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.269889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4t9\" (UniqueName: \"kubernetes.io/projected/bfcc8a43-9aab-47d9-938f-76f2e6ea1c97-kube-api-access-pj4t9\") pod \"managed-serviceaccount-addon-agent-845656c4c4-zptcl\" (UID: \"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.270117 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.269947    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfcc8a43-9aab-47d9-938f-76f2e6ea1c97-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-845656c4c4-zptcl\" (UID: \"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.304193 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.304160    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-795764c55d-qhqkl"]
Sep 17 17:42:31.307326 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.307307    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.333291 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.333272    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795764c55d-qhqkl"]
Sep 17 17:42:31.371271 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfcc8a43-9aab-47d9-938f-76f2e6ea1c97-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-845656c4c4-zptcl\" (UID: \"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.371379 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjfq\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-kube-api-access-psjfq\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371473 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371455    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-registry-tls\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371527 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371482    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4bc3f6a-8dce-46c6-b793-947b2469763e-installation-pull-secrets\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371527 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4bc3f6a-8dce-46c6-b793-947b2469763e-ca-trust-extracted\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371527 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371518    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4bc3f6a-8dce-46c6-b793-947b2469763e-registry-certificates\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371683 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371590    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4bc3f6a-8dce-46c6-b793-947b2469763e-trusted-ca\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371683 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371640    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4t9\" (UniqueName: \"kubernetes.io/projected/bfcc8a43-9aab-47d9-938f-76f2e6ea1c97-kube-api-access-pj4t9\") pod \"managed-serviceaccount-addon-agent-845656c4c4-zptcl\" (UID: \"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.371683 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371667    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-bound-sa-token\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.371850 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.371726    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4bc3f6a-8dce-46c6-b793-947b2469763e-image-registry-private-configuration\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.373934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.373916    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfcc8a43-9aab-47d9-938f-76f2e6ea1c97-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-845656c4c4-zptcl\" (UID: \"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.385559 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.385540    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4t9\" (UniqueName: \"kubernetes.io/projected/bfcc8a43-9aab-47d9-938f-76f2e6ea1c97-kube-api-access-pj4t9\") pod \"managed-serviceaccount-addon-agent-845656c4c4-zptcl\" (UID: \"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.472253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-registry-tls\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4bc3f6a-8dce-46c6-b793-947b2469763e-installation-pull-secrets\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472402 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472252    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4bc3f6a-8dce-46c6-b793-947b2469763e-ca-trust-extracted\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472402 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4bc3f6a-8dce-46c6-b793-947b2469763e-registry-certificates\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472402 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472303    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4bc3f6a-8dce-46c6-b793-947b2469763e-trusted-ca\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472402 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472338    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-bound-sa-token\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472402 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4bc3f6a-8dce-46c6-b793-947b2469763e-image-registry-private-configuration\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.472931 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.472903    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjfq\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-kube-api-access-psjfq\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.473160 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.473116    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4bc3f6a-8dce-46c6-b793-947b2469763e-ca-trust-extracted\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.473899 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.473872    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4bc3f6a-8dce-46c6-b793-947b2469763e-registry-certificates\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.474403 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.474377    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4bc3f6a-8dce-46c6-b793-947b2469763e-trusted-ca\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.476713 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.476688    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4bc3f6a-8dce-46c6-b793-947b2469763e-image-registry-private-configuration\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.477963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.477945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-registry-tls\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.479590 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.479572    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4bc3f6a-8dce-46c6-b793-947b2469763e-installation-pull-secrets\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.483509 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.483489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-bound-sa-token\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.484637 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.484619    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjfq\" (UniqueName: \"kubernetes.io/projected/a4bc3f6a-8dce-46c6-b793-947b2469763e-kube-api-access-psjfq\") pod \"image-registry-795764c55d-qhqkl\" (UID: \"a4bc3f6a-8dce-46c6-b793-947b2469763e\") " pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.533441 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.533418    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"
Sep 17 17:42:31.615994 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.615966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:31.800513 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.800491    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl"]
Sep 17 17:42:31.802370 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:31.802349    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfcc8a43_9aab_47d9_938f_76f2e6ea1c97.slice/crio-2ddcc6a2b00123a9a90442deaac066a2be56031202fdd6de4505d361770cf318 WatchSource:0}: Error finding container 2ddcc6a2b00123a9a90442deaac066a2be56031202fdd6de4505d361770cf318: Status 404 returned error can't find the container with id 2ddcc6a2b00123a9a90442deaac066a2be56031202fdd6de4505d361770cf318
Sep 17 17:42:31.886651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:31.886628    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795764c55d-qhqkl"]
Sep 17 17:42:31.888597 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:31.888576    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bc3f6a_8dce_46c6_b793_947b2469763e.slice/crio-374880eaab8f5eb230b8001778fcc3d764e3d22c8130079837a0df0a36f91d7d WatchSource:0}: Error finding container 374880eaab8f5eb230b8001778fcc3d764e3d22c8130079837a0df0a36f91d7d: Status 404 returned error can't find the container with id 374880eaab8f5eb230b8001778fcc3d764e3d22c8130079837a0df0a36f91d7d
Sep 17 17:42:32.294733 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:32.294698    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795764c55d-qhqkl" event={"ID":"a4bc3f6a-8dce-46c6-b793-947b2469763e","Type":"ContainerStarted","Data":"9da2d4e67584274b08830f3cc88387a151bb9b1ffe3d2ee6534095f91188b41d"}
Sep 17 17:42:32.295223 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:32.294742    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795764c55d-qhqkl" event={"ID":"a4bc3f6a-8dce-46c6-b793-947b2469763e","Type":"ContainerStarted","Data":"374880eaab8f5eb230b8001778fcc3d764e3d22c8130079837a0df0a36f91d7d"}
Sep 17 17:42:32.295223 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:32.294771    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:32.295865 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:32.295841    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl" event={"ID":"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97","Type":"ContainerStarted","Data":"2ddcc6a2b00123a9a90442deaac066a2be56031202fdd6de4505d361770cf318"}
Sep 17 17:42:32.322608 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:32.322519    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-795764c55d-qhqkl" podStartSLOduration=1.322499547 podStartE2EDuration="1.322499547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:32.321514996 +0000 UTC m=+85.512852680" watchObservedRunningTime="2026-09-17 17:42:32.322499547 +0000 UTC m=+85.513837225"
Sep 17 17:42:34.304872 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:34.304787    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl" event={"ID":"bfcc8a43-9aab-47d9-938f-76f2e6ea1c97","Type":"ContainerStarted","Data":"37d8a40cd325b12b2d8eb8d1b17e3fe584ac3b99f1ed79aa6a55469872e7c831"}
Sep 17 17:42:34.323724 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:34.323677    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-845656c4c4-zptcl" podStartSLOduration=1.087266483 podStartE2EDuration="3.32366212s" totalImagesPullingTime="2.236395637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:34.322250424 +0000 UTC m=+87.513588110" watchObservedRunningTime="2026-09-17 17:42:34.32366212 +0000 UTC m=+87.515000188"
Sep 17 17:42:36.712064 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:36.712031    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:36.714364 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:36.714337    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbcjh\" (UID: \"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9\") " pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:36.727747 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:36.727721    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cb5pj\""
Sep 17 17:42:36.736063 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:36.736037    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbcjh"
Sep 17 17:42:37.026986 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:37.026889    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbcjh"]
Sep 17 17:42:37.030561 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:37.030534    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0491fc_6fa7_48ef_9fac_85bc0e06a6f9.slice/crio-92382795afb2b762637e6758831e9ea0d958d88c45d0b8ed63a01df19c5d6eab WatchSource:0}: Error finding container 92382795afb2b762637e6758831e9ea0d958d88c45d0b8ed63a01df19c5d6eab: Status 404 returned error can't find the container with id 92382795afb2b762637e6758831e9ea0d958d88c45d0b8ed63a01df19c5d6eab
Sep 17 17:42:37.315928 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:37.315851    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbcjh" event={"ID":"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9","Type":"ContainerStarted","Data":"46bb089cffc1e77afacd3eba4f50d161af59f880b87452709ed629c5e4a43b00"}
Sep 17 17:42:37.315928 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:37.315905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbcjh" event={"ID":"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9","Type":"ContainerStarted","Data":"92382795afb2b762637e6758831e9ea0d958d88c45d0b8ed63a01df19c5d6eab"}
Sep 17 17:42:38.321589 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:38.321553    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbcjh" event={"ID":"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9","Type":"ContainerStarted","Data":"c580b6489ed77f0c2abaa8d344fda278dad750e2339eb9b622832670a9c1f9a6"}
Sep 17 17:42:39.326364 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:39.326295    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbcjh" event={"ID":"1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9","Type":"ContainerStarted","Data":"65e5114e9f433b04c5c40fd8e98bd60cfb65ff7f2a8de7722a1a0d72d52f24d9"}
Sep 17 17:42:39.346511 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:39.346467    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zbcjh" podStartSLOduration=33.505609778 podStartE2EDuration="35.346452733s" totalImagesPullingTime="1.840842955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:39.346128378 +0000 UTC m=+92.537466064" watchObservedRunningTime="2026-09-17 17:42:39.346452733 +0000 UTC m=+92.537790418"
Sep 17 17:42:41.228403 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:41.228373    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:48.483264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.483223    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"]
Sep 17 17:42:48.486297 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.486271    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-8785r"]
Sep 17 17:42:48.486446 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.486411    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.489222 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.489200    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.489339 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.489220    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:42:48.489619 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.489602    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:42:48.497313 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.497293    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:42:48.497521 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.497310    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:42:48.497619 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.497398    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:42:48.498656 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.498641    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:42:48.498830 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.498791    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:42:48.499061 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.498898    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lnmqq\""
Sep 17 17:42:48.499133 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.498932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6dmpv\""
Sep 17 17:42:48.499133 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.498963    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:42:48.503880 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.503639    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"]
Sep 17 17:42:48.505068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.505047    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-8785r"]
Sep 17 17:42:48.525363 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.525343    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-86mjv"]
Sep 17 17:42:48.528885 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.528867    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.531603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.531587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:48.531855 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.531839    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvws2\""
Sep 17 17:42:48.531980 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.531898    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:48.532093 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.532078    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:48.594443 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594415    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.594594 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594456    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt86j\" (UniqueName: \"kubernetes.io/projected/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-api-access-vt86j\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.594594 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.594594 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594519    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-tls\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.594786 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-sys\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.594786 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.594786 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.594786 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594725    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f2cd82-a372-4402-907f-21ee068db39d-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.594786 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-textfile\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594852    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-root\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594888    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fdb8f115-24a5-4243-a106-3da5baf1f912-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594918    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8f115-24a5-4243-a106-3da5baf1f912-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-wtmp\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.594998    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.595046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.595025    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.595299 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.595052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxbw\" (UniqueName: \"kubernetes.io/projected/e8f2cd82-a372-4402-907f-21ee068db39d-kube-api-access-6rxbw\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.595299 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.595073    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-metrics-client-ca\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.595299 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.595105    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2cc\" (UniqueName: \"kubernetes.io/projected/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-kube-api-access-zr2cc\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.695572 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695544    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fdb8f115-24a5-4243-a106-3da5baf1f912-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.695572 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8f115-24a5-4243-a106-3da5baf1f912-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.695752 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.695752 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-wtmp\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.695752 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695627    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.695752 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695651    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.695752 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695676    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxbw\" (UniqueName: \"kubernetes.io/projected/e8f2cd82-a372-4402-907f-21ee068db39d-kube-api-access-6rxbw\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.696027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-metrics-client-ca\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2cc\" (UniqueName: \"kubernetes.io/projected/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-kube-api-access-zr2cc\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695883    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.696027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695907    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-wtmp\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.695914    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt86j\" (UniqueName: \"kubernetes.io/projected/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-api-access-vt86j\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.696027 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:48.696010    2585 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:48.696077    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-tls podName:fdb8f115-24a5-4243-a106-3da5baf1f912 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:49.196057976 +0000 UTC m=+102.387395653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-8785r" (UID: "fdb8f115-24a5-4243-a106-3da5baf1f912") : secret "kube-state-metrics-tls" not found
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-tls\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-sys\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696273    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.696312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f2cd82-a372-4402-907f-21ee068db39d-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.696658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696321    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-textfile\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-root\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-root\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.696658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696644    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fdb8f115-24a5-4243-a106-3da5baf1f912-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.696870 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:48.696749    2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 17:42:48.696870 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:42:48.696806    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-tls podName:e8f2cd82-a372-4402-907f-21ee068db39d nodeName:}" failed. No retries permitted until 2026-09-17 17:42:49.196790185 +0000 UTC m=+102.388127862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-4jcgh" (UID: "e8f2cd82-a372-4402-907f-21ee068db39d") : secret "openshift-state-metrics-tls" not found
Sep 17 17:42:48.696977 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.696876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-sys\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.697117 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.697092    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-metrics-client-ca\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.697538 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.697519    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.697927 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.697618    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.697927 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.697749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f2cd82-a372-4402-907f-21ee068db39d-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.697927 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.697893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-textfile\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.697927 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.697920    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8f115-24a5-4243-a106-3da5baf1f912-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.699483 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.699077    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-tls\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.699483 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.699100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.699954 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.699936    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.700256 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.700237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.712294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.712260    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2cc\" (UniqueName: \"kubernetes.io/projected/de540b4a-d1bb-474f-b327-a9c7b0cd67a5-kube-api-access-zr2cc\") pod \"node-exporter-86mjv\" (UID: \"de540b4a-d1bb-474f-b327-a9c7b0cd67a5\") " pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.715766 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.715744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxbw\" (UniqueName: \"kubernetes.io/projected/e8f2cd82-a372-4402-907f-21ee068db39d-kube-api-access-6rxbw\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:48.717913 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.717892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt86j\" (UniqueName: \"kubernetes.io/projected/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-api-access-vt86j\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:48.839100 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:48.839032    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86mjv"
Sep 17 17:42:48.851772 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:48.851746    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde540b4a_d1bb_474f_b327_a9c7b0cd67a5.slice/crio-4a803034b18ca754caf2bec21b02cbfa1c0b50f9f67623b10415e9ee153c745d WatchSource:0}: Error finding container 4a803034b18ca754caf2bec21b02cbfa1c0b50f9f67623b10415e9ee153c745d: Status 404 returned error can't find the container with id 4a803034b18ca754caf2bec21b02cbfa1c0b50f9f67623b10415e9ee153c745d
Sep 17 17:42:49.200163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.200129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:49.200329 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.200179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:49.202882 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.202856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2cd82-a372-4402-907f-21ee068db39d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-4jcgh\" (UID: \"e8f2cd82-a372-4402-907f-21ee068db39d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:49.202976 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.202938    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fdb8f115-24a5-4243-a106-3da5baf1f912-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-8785r\" (UID: \"fdb8f115-24a5-4243-a106-3da5baf1f912\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:49.359276 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.359231    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86mjv" event={"ID":"de540b4a-d1bb-474f-b327-a9c7b0cd67a5","Type":"ContainerStarted","Data":"4a803034b18ca754caf2bec21b02cbfa1c0b50f9f67623b10415e9ee153c745d"}
Sep 17 17:42:49.398668 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.398641    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"
Sep 17 17:42:49.407507 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.407479    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r"
Sep 17 17:42:49.499982 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.499947    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:42:49.505076 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.505032    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.509944 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.509886    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ssm5f\""
Sep 17 17:42:49.510168 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.510148    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:42:49.510386 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.510369    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:42:49.511641 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.511623    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:42:49.512121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.511937    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:42:49.512205 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.512132    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:42:49.512884 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.512863    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:42:49.513091 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.513072    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:42:49.514405 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.514384    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:42:49.514637 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.514616    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:42:49.575256 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.575226    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:42:49.603146 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603117    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-out\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603303 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603171    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-web-config\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603303 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603242    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603303 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603279    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603334    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603355    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603376    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603429    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603457    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603523    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kvp7\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-kube-api-access-2kvp7\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.603622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.603615    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.690289 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.690264    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh"]
Sep 17 17:42:49.692001 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.691980    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-8785r"]
Sep 17 17:42:49.704494 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-out\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704602 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-web-config\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704602 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704559    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704602 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704584    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704674    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704700    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704726    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.704760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.705068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.705068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704790    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kvp7\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-kube-api-access-2kvp7\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.705068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.704842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.705206 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.705191    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.706318 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.706277    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.707400 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.707375    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.708151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.708129    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-web-config\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.708277 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.708214    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-out\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.708622 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.708599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.712162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.710520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.712162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.710780    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.712162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.711125    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.712162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.711417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.712162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.711958    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.712162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.711958    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.724838 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.724797    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kvp7\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-kube-api-access-2kvp7\") pod \"alertmanager-main-0\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:49.813157 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:49.813123    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb8f115_24a5_4243_a106_3da5baf1f912.slice/crio-5166d4424c76c2dadf0a72cbaec9bd0d5b9448b3d8fa27e95deac14e6f4570c7 WatchSource:0}: Error finding container 5166d4424c76c2dadf0a72cbaec9bd0d5b9448b3d8fa27e95deac14e6f4570c7: Status 404 returned error can't find the container with id 5166d4424c76c2dadf0a72cbaec9bd0d5b9448b3d8fa27e95deac14e6f4570c7
Sep 17 17:42:49.813502 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:49.813479    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f2cd82_a372_4402_907f_21ee068db39d.slice/crio-c1a1c0f974604ee55ba58e8406bae613b4d8b3ad57a455e678e3a5e59faab566 WatchSource:0}: Error finding container c1a1c0f974604ee55ba58e8406bae613b4d8b3ad57a455e678e3a5e59faab566: Status 404 returned error can't find the container with id c1a1c0f974604ee55ba58e8406bae613b4d8b3ad57a455e678e3a5e59faab566
Sep 17 17:42:49.818740 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:49.818714    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:50.105756 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.105727    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:42:50.107395 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:42:50.107372    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8305d83b_c6b5_4f04_a677_34b4dbe9f1fb.slice/crio-ab347f8701eb43fb2acdc9fa9ea8b788af2028fd9e14c2ccd035e2f0f8b5ce01 WatchSource:0}: Error finding container ab347f8701eb43fb2acdc9fa9ea8b788af2028fd9e14c2ccd035e2f0f8b5ce01: Status 404 returned error can't find the container with id ab347f8701eb43fb2acdc9fa9ea8b788af2028fd9e14c2ccd035e2f0f8b5ce01
Sep 17 17:42:50.364456 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.364379    2585 generic.go:356] "Generic (PLEG): container finished" podID="de540b4a-d1bb-474f-b327-a9c7b0cd67a5" containerID="688f15975d877f455aa4c1f84224a4b667008dc7e1c2fe28ba6d6cd753f2201d" exitCode=0
Sep 17 17:42:50.364675 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.364650    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86mjv" event={"ID":"de540b4a-d1bb-474f-b327-a9c7b0cd67a5","Type":"ContainerDied","Data":"688f15975d877f455aa4c1f84224a4b667008dc7e1c2fe28ba6d6cd753f2201d"}
Sep 17 17:42:50.366879 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.366845    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh" event={"ID":"e8f2cd82-a372-4402-907f-21ee068db39d","Type":"ContainerStarted","Data":"0ff6367fdc0ae1811712cfd1803c2c0f3524678140f9a862ab9e785ebf61ef75"}
Sep 17 17:42:50.366976 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.366918    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh" event={"ID":"e8f2cd82-a372-4402-907f-21ee068db39d","Type":"ContainerStarted","Data":"2850107bded3d6637cc16f9297098db81d07682a6e65a83852e3dce6e312ec72"}
Sep 17 17:42:50.366976 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.366948    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh" event={"ID":"e8f2cd82-a372-4402-907f-21ee068db39d","Type":"ContainerStarted","Data":"c1a1c0f974604ee55ba58e8406bae613b4d8b3ad57a455e678e3a5e59faab566"}
Sep 17 17:42:50.369540 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.368694    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"ab347f8701eb43fb2acdc9fa9ea8b788af2028fd9e14c2ccd035e2f0f8b5ce01"}
Sep 17 17:42:50.370185 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:50.370156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r" event={"ID":"fdb8f115-24a5-4243-a106-3da5baf1f912","Type":"ContainerStarted","Data":"5166d4424c76c2dadf0a72cbaec9bd0d5b9448b3d8fa27e95deac14e6f4570c7"}
Sep 17 17:42:51.332121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.332084    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"]
Sep 17 17:42:51.346562 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.346515    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.349106 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349077    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"]
Sep 17 17:42:51.349286 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349272    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:42:51.349423 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349402    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v6h75\""
Sep 17 17:42:51.349489 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349448    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:42:51.349527 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349516    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:42:51.349590 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349574    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:42:51.349635 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349402    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:42:51.349726 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.349712    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ekn29sej7cao9\""
Sep 17 17:42:51.378562 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.378477    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86mjv" event={"ID":"de540b4a-d1bb-474f-b327-a9c7b0cd67a5","Type":"ContainerStarted","Data":"6fb3561a390f6bfc1942e0b1444474e15b2432e8005471bb1aa6b9fd2f24b32f"}
Sep 17 17:42:51.378562 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.378531    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86mjv" event={"ID":"de540b4a-d1bb-474f-b327-a9c7b0cd67a5","Type":"ContainerStarted","Data":"a8cb5590cd07c35f11b22b294ba1e74404c8e17bd0dee2cb1ec43e3918045543"}
Sep 17 17:42:51.404152 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.404019    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-86mjv" podStartSLOduration=2.325012662 podStartE2EDuration="3.404001806s" totalImagesPullingTime="1.007587744s" totalInitContainerRuntime="71.4014ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:51.403078752 +0000 UTC m=+104.594416436" watchObservedRunningTime="2026-09-17 17:42:51.404001806 +0000 UTC m=+104.595339491"
Sep 17 17:42:51.421905 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.421880    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.421923    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqq7q\" (UniqueName: \"kubernetes.io/projected/3e309d14-c79c-4ea2-b258-319bca8c92c2-kube-api-access-nqq7q\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.421952    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-tls\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422102 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.422035    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422102 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.422072    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422102 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.422093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-grpc-tls\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.422171    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.422253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.422198    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e309d14-c79c-4ea2-b258-319bca8c92c2-metrics-client-ca\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523331 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523452 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523358    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqq7q\" (UniqueName: \"kubernetes.io/projected/3e309d14-c79c-4ea2-b258-319bca8c92c2-kube-api-access-nqq7q\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523537 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-tls\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523660 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523660 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523640    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523759 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523675    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-grpc-tls\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523759 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.523885 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.523766    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e309d14-c79c-4ea2-b258-319bca8c92c2-metrics-client-ca\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.524572 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.524523    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e309d14-c79c-4ea2-b258-319bca8c92c2-metrics-client-ca\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.526323 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.526274    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-tls\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.526448 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.526427    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.526550 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.526530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.526657 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.526640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.526750 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.526734    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-grpc-tls\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.526893 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.526871    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e309d14-c79c-4ea2-b258-319bca8c92c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.531643 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.531620    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqq7q\" (UniqueName: \"kubernetes.io/projected/3e309d14-c79c-4ea2-b258-319bca8c92c2-kube-api-access-nqq7q\") pod \"thanos-querier-55bdbbcdc9-6w5m9\" (UID: \"3e309d14-c79c-4ea2-b258-319bca8c92c2\") " pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:51.659669 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:51.659644    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:52.265919 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:52.265892    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kt64l"
Sep 17 17:42:52.657516 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:52.652263    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"]
Sep 17 17:42:53.304660 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.304634    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-795764c55d-qhqkl"
Sep 17 17:42:53.390392 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.390280    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b" exitCode=0
Sep 17 17:42:53.390392 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.390356    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"}
Sep 17 17:42:53.394087 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.393530    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r" event={"ID":"fdb8f115-24a5-4243-a106-3da5baf1f912","Type":"ContainerStarted","Data":"4ff8661b0756f04239b69ad910e044dd98602b9390ce10d3ece0a2d31ab53418"}
Sep 17 17:42:53.394087 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.393578    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r" event={"ID":"fdb8f115-24a5-4243-a106-3da5baf1f912","Type":"ContainerStarted","Data":"3971a2de240d9da30192800fc186296011f909d40f80fe6544f2f2aa92b97aea"}
Sep 17 17:42:53.394087 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.393599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r" event={"ID":"fdb8f115-24a5-4243-a106-3da5baf1f912","Type":"ContainerStarted","Data":"586bc2537398b113c4f84fa342ad368ac07dd9efe8507d08c33c25c9522774b6"}
Sep 17 17:42:53.395330 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.395269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"154f998cc7cb06c263d0017eda7f921692416ebe2cfeb03c6aa05a847b6b7d1c"}
Sep 17 17:42:53.398084 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.398060    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh" event={"ID":"e8f2cd82-a372-4402-907f-21ee068db39d","Type":"ContainerStarted","Data":"bc895ac360dbde61310d7fec5361835281fb5933d1b74916dec78d6d17c294a6"}
Sep 17 17:42:53.448876 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.448801    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-8785r" podStartSLOduration=2.89692396 podStartE2EDuration="5.44878115s" totalImagesPullingTime="2.55185719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:53.445844532 +0000 UTC m=+106.637182212" watchObservedRunningTime="2026-09-17 17:42:53.44878115 +0000 UTC m=+106.640118838"
Sep 17 17:42:53.467791 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:53.467745    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-4jcgh" podStartSLOduration=3.056259008 podStartE2EDuration="5.46772986s" totalImagesPullingTime="2.411470852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:53.466377457 +0000 UTC m=+106.657715124" watchObservedRunningTime="2026-09-17 17:42:53.46772986 +0000 UTC m=+106.659067544"
Sep 17 17:42:55.409559 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:55.409534    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"}
Sep 17 17:42:55.411709 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:55.411562    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"ae4afe06976777e8f712c7f5bcf256b5d1d3e0abdb41bd1382818f182d513095"}
Sep 17 17:42:55.411709 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:55.411619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"c559238b67cf30d578587f096d6e64f1ba34e2a8f6abda6c903bb4a4043902e3"}
Sep 17 17:42:56.240335 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.240299    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj" podUID="4dc6558d-93df-4e43-99dd-a49e53ad28df" containerName="registry" containerID="cri-o://03c865ae0a45589615ae587bfd1c293de91f783d81fbb26018792060cd17ab54" gracePeriod=30
Sep 17 17:42:56.418429 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.418397    2585 generic.go:356] "Generic (PLEG): container finished" podID="4dc6558d-93df-4e43-99dd-a49e53ad28df" containerID="03c865ae0a45589615ae587bfd1c293de91f783d81fbb26018792060cd17ab54" exitCode=0
Sep 17 17:42:56.418800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.418499    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj" event={"ID":"4dc6558d-93df-4e43-99dd-a49e53ad28df","Type":"ContainerDied","Data":"03c865ae0a45589615ae587bfd1c293de91f783d81fbb26018792060cd17ab54"}
Sep 17 17:42:56.422709 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.422676    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"}
Sep 17 17:42:56.422833 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.422752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"}
Sep 17 17:42:56.422833 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.422797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"}
Sep 17 17:42:56.422962 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.422868    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"}
Sep 17 17:42:56.422962 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.422910    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerStarted","Data":"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"}
Sep 17 17:42:56.427620 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.427586    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"112972ced0402f48d5da1d02664c944f1425c24357ad09f5ce8535d349eee293"}
Sep 17 17:42:56.427731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.427644    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"8f2af316263e340bbc6002f97b668321fcd22ba3aea1335d46db42af8412fae3"}
Sep 17 17:42:56.427731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.427669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"439e72fb1e2f2999b31e7ddd9dd72cb74b09faf69895351b7231da2fdae4a66d"}
Sep 17 17:42:56.427731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.427697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" event={"ID":"3e309d14-c79c-4ea2-b258-319bca8c92c2","Type":"ContainerStarted","Data":"8ad11a1f9c8a279949a28b8b3416167b32a6c194578edff7568fb61555714128"}
Sep 17 17:42:56.427731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.427726    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:42:56.460716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.460671    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.858159865 podStartE2EDuration="7.460657889s" totalImagesPullingTime="4.56318022s" totalInitContainerRuntime="39.317804ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:56.4594852 +0000 UTC m=+109.650822886" watchObservedRunningTime="2026-09-17 17:42:56.460657889 +0000 UTC m=+109.651995574"
Sep 17 17:42:56.488769 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.488750    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:56.491895 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.491858    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9" podStartSLOduration=2.320859662 podStartE2EDuration="5.491844361s" totalImagesPullingTime="3.170984699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:56.489745558 +0000 UTC m=+109.681083272" watchObservedRunningTime="2026-09-17 17:42:56.491844361 +0000 UTC m=+109.683182046"
Sep 17 17:42:56.671686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-trusted-ca\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.671686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671641    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.671686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671660    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-installation-pull-secrets\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.671686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671674    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-bound-sa-token\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.672048 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671701    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-image-registry-private-configuration\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.672048 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671759    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v95r\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-kube-api-access-8v95r\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.672048 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671801    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-certificates\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.672048 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.671881    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dc6558d-93df-4e43-99dd-a49e53ad28df-ca-trust-extracted\") pod \"4dc6558d-93df-4e43-99dd-a49e53ad28df\" (UID: \"4dc6558d-93df-4e43-99dd-a49e53ad28df\") "
Sep 17 17:42:56.672228 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.672130    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:56.672379 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.672355    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:56.674567 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.674541    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:56.674685 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.674595    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:56.674750 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.674681    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:56.674750 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.674711    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:56.674907 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.674891    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-kube-api-access-8v95r" (OuterVolumeSpecName: "kube-api-access-8v95r") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "kube-api-access-8v95r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:56.680254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.680235    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc6558d-93df-4e43-99dd-a49e53ad28df-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4dc6558d-93df-4e43-99dd-a49e53ad28df" (UID: "4dc6558d-93df-4e43-99dd-a49e53ad28df"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:56.772595 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772571    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v95r\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-kube-api-access-8v95r\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772595 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772593    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-certificates\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772602    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dc6558d-93df-4e43-99dd-a49e53ad28df-ca-trust-extracted\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772612    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6558d-93df-4e43-99dd-a49e53ad28df-trusted-ca\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772620    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-registry-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772628    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-installation-pull-secrets\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772636    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dc6558d-93df-4e43-99dd-a49e53ad28df-bound-sa-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.772730 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:56.772644    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dc6558d-93df-4e43-99dd-a49e53ad28df-image-registry-private-configuration\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:42:57.432409 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:57.432383    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj"
Sep 17 17:42:57.432872 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:57.432423    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff67d7dcd-htsxj" event={"ID":"4dc6558d-93df-4e43-99dd-a49e53ad28df","Type":"ContainerDied","Data":"276dba731937a4637174e84d85b0628abebd1a54ae2ea18b6b7188b6041550ce"}
Sep 17 17:42:57.432872 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:57.432533    2585 scope.go:117] "RemoveContainer" containerID="03c865ae0a45589615ae587bfd1c293de91f783d81fbb26018792060cd17ab54"
Sep 17 17:42:57.473196 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:57.473171    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff67d7dcd-htsxj"]
Sep 17 17:42:57.476792 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:57.476760    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-ff67d7dcd-htsxj"]
Sep 17 17:42:58.958851 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:42:58.958805    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc6558d-93df-4e43-99dd-a49e53ad28df" path="/var/lib/kubelet/pods/4dc6558d-93df-4e43-99dd-a49e53ad28df/volumes"
Sep 17 17:43:02.070019 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.069984    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-zr5kx"]
Sep 17 17:43:02.070406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.070294    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc6558d-93df-4e43-99dd-a49e53ad28df" containerName="registry"
Sep 17 17:43:02.070406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.070307    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc6558d-93df-4e43-99dd-a49e53ad28df" containerName="registry"
Sep 17 17:43:02.070406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.070372    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc6558d-93df-4e43-99dd-a49e53ad28df" containerName="registry"
Sep 17 17:43:02.075383 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.075365    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:02.078187 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.078155    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:43:02.078292 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.078248    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:43:02.078292 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.078261    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2q5jh\""
Sep 17 17:43:02.088604 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.088586    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-zr5kx"]
Sep 17 17:43:02.213484 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.213455    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b648g\" (UniqueName: \"kubernetes.io/projected/5008d57f-0a28-4cfe-a37f-08667f53ea6e-kube-api-access-b648g\") pod \"downloads-6594f4c75-zr5kx\" (UID: \"5008d57f-0a28-4cfe-a37f-08667f53ea6e\") " pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:02.314624 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.314594    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b648g\" (UniqueName: \"kubernetes.io/projected/5008d57f-0a28-4cfe-a37f-08667f53ea6e-kube-api-access-b648g\") pod \"downloads-6594f4c75-zr5kx\" (UID: \"5008d57f-0a28-4cfe-a37f-08667f53ea6e\") " pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:02.323526 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.323470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b648g\" (UniqueName: \"kubernetes.io/projected/5008d57f-0a28-4cfe-a37f-08667f53ea6e-kube-api-access-b648g\") pod \"downloads-6594f4c75-zr5kx\" (UID: \"5008d57f-0a28-4cfe-a37f-08667f53ea6e\") " pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:02.383937 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.383904    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:02.441889 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.441863    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55bdbbcdc9-6w5m9"
Sep 17 17:43:02.652017 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:02.651985    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-zr5kx"]
Sep 17 17:43:02.655252 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:43:02.655227    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5008d57f_0a28_4cfe_a37f_08667f53ea6e.slice/crio-e74fd33253293e37f9f4254fc36caddf21d0366ae0a40860bc76173dea6f86e4 WatchSource:0}: Error finding container e74fd33253293e37f9f4254fc36caddf21d0366ae0a40860bc76173dea6f86e4: Status 404 returned error can't find the container with id e74fd33253293e37f9f4254fc36caddf21d0366ae0a40860bc76173dea6f86e4
Sep 17 17:43:03.461905 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:03.461865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-zr5kx" event={"ID":"5008d57f-0a28-4cfe-a37f-08667f53ea6e","Type":"ContainerStarted","Data":"e74fd33253293e37f9f4254fc36caddf21d0366ae0a40860bc76173dea6f86e4"}
Sep 17 17:43:15.984287 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.984246    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb47b8c79-wfwlp"]
Sep 17 17:43:15.988586 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.988558    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:15.993006 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.992978    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:43:15.993141 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.993020    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:43:15.993141 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.993067    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p28vv\""
Sep 17 17:43:15.993141 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.993018    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:43:15.993141 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.992978    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:43:15.993368 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.993350    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:43:15.997633 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.997613    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:43:15.999915 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:15.999897    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb47b8c79-wfwlp"]
Sep 17 17:43:16.039523 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-oauth-config\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.039523 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039517    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxqq\" (UniqueName: \"kubernetes.io/projected/999a6c2e-bed4-4c07-8163-d36b2037955b-kube-api-access-ppxqq\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.039713 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039596    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-service-ca\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.039713 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039644    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-serving-cert\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.039713 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-oauth-serving-cert\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.039878 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039777    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-console-config\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.039878 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.039834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-trusted-ca-bundle\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.140923 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.140895    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-service-ca\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141077 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.140937    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-serving-cert\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141077 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.140952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-oauth-serving-cert\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141201 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-console-config\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141201 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141118    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-trusted-ca-bundle\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141333 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141291    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-oauth-config\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141469 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxqq\" (UniqueName: \"kubernetes.io/projected/999a6c2e-bed4-4c07-8163-d36b2037955b-kube-api-access-ppxqq\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141631    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-service-ca\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141796 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141678    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-oauth-serving-cert\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141796 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141718    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-console-config\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.141999 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.141981    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-trusted-ca-bundle\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.144202 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.144179    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-oauth-config\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.144400 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.144376    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-serving-cert\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.150318 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.150293    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxqq\" (UniqueName: \"kubernetes.io/projected/999a6c2e-bed4-4c07-8163-d36b2037955b-kube-api-access-ppxqq\") pod \"console-7bb47b8c79-wfwlp\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") " pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:16.299911 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:16.299805    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:18.766305 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:18.766276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:43:18.768491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:18.768470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5-metrics-certs\") pod \"network-metrics-daemon-29686\" (UID: \"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5\") " pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:43:18.951269 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:18.951182    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb47b8c79-wfwlp"]
Sep 17 17:43:18.978803 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:18.978717    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nqf6f\""
Sep 17 17:43:18.986329 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:18.986179    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29686"
Sep 17 17:43:19.260822 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.260778    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29686"]
Sep 17 17:43:19.263472 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:43:19.263441    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda842e3d3_81ce_4a5a_bbd5_12edef8ff6e5.slice/crio-5ed2e596281cbe6557ce5cba3421464a0d544ff159cbb4764fa12555f81cec44 WatchSource:0}: Error finding container 5ed2e596281cbe6557ce5cba3421464a0d544ff159cbb4764fa12555f81cec44: Status 404 returned error can't find the container with id 5ed2e596281cbe6557ce5cba3421464a0d544ff159cbb4764fa12555f81cec44
Sep 17 17:43:19.545638 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.545554    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-zr5kx" event={"ID":"5008d57f-0a28-4cfe-a37f-08667f53ea6e","Type":"ContainerStarted","Data":"f8da7377b2d73d7308e939cf09ca3c79646eb52915501c25bd8f98d92879fd70"}
Sep 17 17:43:19.546190 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.546171    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:19.548686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.548657    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb47b8c79-wfwlp" event={"ID":"999a6c2e-bed4-4c07-8163-d36b2037955b","Type":"ContainerStarted","Data":"c2cc53b1f69d95e1a7df0cd56584ab712f6f831e8e88fa8c4df009537e33ce35"}
Sep 17 17:43:19.550644 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.550597    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29686" event={"ID":"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5","Type":"ContainerStarted","Data":"5ed2e596281cbe6557ce5cba3421464a0d544ff159cbb4764fa12555f81cec44"}
Sep 17 17:43:19.557971 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.557944    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-zr5kx"
Sep 17 17:43:19.569093 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:19.569038    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-zr5kx" podStartSLOduration=1.4663043230000001 podStartE2EDuration="17.569020286s" totalImagesPullingTime="16.102715963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:19.566516941 +0000 UTC m=+132.757854628" watchObservedRunningTime="2026-09-17 17:43:19.569020286 +0000 UTC m=+132.760357972"
Sep 17 17:43:25.584657 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.584563    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29686" event={"ID":"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5","Type":"ContainerStarted","Data":"4310fd5b82072875344bfd84cff753565ba546a544f7bba7dabd4221ac23ebfd"}
Sep 17 17:43:25.584657 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.584628    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29686" event={"ID":"a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5","Type":"ContainerStarted","Data":"a2fcc3256b3b850e09c6283d1d6003706c51c1857968a6ec78b90916c9c80b1a"}
Sep 17 17:43:25.586582 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.586542    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb47b8c79-wfwlp" event={"ID":"999a6c2e-bed4-4c07-8163-d36b2037955b","Type":"ContainerStarted","Data":"692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f"}
Sep 17 17:43:25.588451 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.588423    2585 generic.go:356] "Generic (PLEG): container finished" podID="29df95df-3880-4cf6-9de8-caa826016278" containerID="eda1457801ae8e798da6e254a8f173ebfc2ad1602548d743e4293941782aa7cf" exitCode=0
Sep 17 17:43:25.588571 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.588463    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx" event={"ID":"29df95df-3880-4cf6-9de8-caa826016278","Type":"ContainerDied","Data":"eda1457801ae8e798da6e254a8f173ebfc2ad1602548d743e4293941782aa7cf"}
Sep 17 17:43:25.588774 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.588758    2585 scope.go:117] "RemoveContainer" containerID="eda1457801ae8e798da6e254a8f173ebfc2ad1602548d743e4293941782aa7cf"
Sep 17 17:43:25.603676 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.603627    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29686" podStartSLOduration=130.965788306 podStartE2EDuration="2m16.603609135s" totalImagesPullingTime="5.637820829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:25.601722369 +0000 UTC m=+138.793060066" watchObservedRunningTime="2026-09-17 17:43:25.603609135 +0000 UTC m=+138.794946823"
Sep 17 17:43:25.637590 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:25.637535    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bb47b8c79-wfwlp" podStartSLOduration=4.301029202 podStartE2EDuration="10.637515388s" totalImagesPullingTime="6.336486186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:25.636908368 +0000 UTC m=+138.828246054" watchObservedRunningTime="2026-09-17 17:43:25.637515388 +0000 UTC m=+138.828853074"
Sep 17 17:43:26.300962 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:26.300924    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:26.301225 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:26.300986    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:26.306662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:26.306631    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:26.595693 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:26.595604    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-892xx" event={"ID":"29df95df-3880-4cf6-9de8-caa826016278","Type":"ContainerStarted","Data":"0dce51092beb689ef7b9c348412869aa6e8fbd167f7555000c36e9f8d8b8fcfb"}
Sep 17 17:43:26.600044 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:26.600021    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:43:30.646076 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:30.646047    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/init-config-reloader/0.log"
Sep 17 17:43:30.845776 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:30.845750    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/alertmanager/0.log"
Sep 17 17:43:31.045731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:31.045655    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/config-reloader/0.log"
Sep 17 17:43:31.246293 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:31.246260    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/kube-rbac-proxy-web/0.log"
Sep 17 17:43:31.445570 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:31.445547    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/kube-rbac-proxy/0.log"
Sep 17 17:43:31.646385 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:31.646357    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/kube-rbac-proxy-metric/0.log"
Sep 17 17:43:31.850640 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:31.850559    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/prom-label-proxy/0.log"
Sep 17 17:43:32.246089 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:32.246057    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-8785r_fdb8f115-24a5-4243-a106-3da5baf1f912/kube-state-metrics/0.log"
Sep 17 17:43:32.445939 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:32.445912    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-8785r_fdb8f115-24a5-4243-a106-3da5baf1f912/kube-rbac-proxy-main/0.log"
Sep 17 17:43:32.646230 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:32.646203    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-8785r_fdb8f115-24a5-4243-a106-3da5baf1f912/kube-rbac-proxy-self/0.log"
Sep 17 17:43:33.845534 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:33.845508    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86mjv_de540b4a-d1bb-474f-b327-a9c7b0cd67a5/init-textfile/0.log"
Sep 17 17:43:34.046880 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:34.046851    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86mjv_de540b4a-d1bb-474f-b327-a9c7b0cd67a5/node-exporter/0.log"
Sep 17 17:43:34.245589 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:34.245564    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86mjv_de540b4a-d1bb-474f-b327-a9c7b0cd67a5/kube-rbac-proxy/0.log"
Sep 17 17:43:35.045759 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:35.045731    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-4jcgh_e8f2cd82-a372-4402-907f-21ee068db39d/kube-rbac-proxy-main/0.log"
Sep 17 17:43:35.245878 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:35.245848    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-4jcgh_e8f2cd82-a372-4402-907f-21ee068db39d/kube-rbac-proxy-self/0.log"
Sep 17 17:43:35.446687 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:35.446655    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-4jcgh_e8f2cd82-a372-4402-907f-21ee068db39d/openshift-state-metrics/0.log"
Sep 17 17:43:37.646634 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:37.646605    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/thanos-query/0.log"
Sep 17 17:43:37.846179 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:37.846147    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy-web/0.log"
Sep 17 17:43:38.045241 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:38.045153    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy/0.log"
Sep 17 17:43:38.246008 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:38.245980    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/prom-label-proxy/0.log"
Sep 17 17:43:38.445639 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:38.445615    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy-rules/0.log"
Sep 17 17:43:38.645581 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:38.645554    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy-metrics/0.log"
Sep 17 17:43:39.446100 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:39.446074    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bb47b8c79-wfwlp_999a6c2e-bed4-4c07-8163-d36b2037955b/console/0.log"
Sep 17 17:43:39.848014 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:39.847933    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-zr5kx_5008d57f-0a28-4cfe-a37f-08667f53ea6e/download-server/0.log"
Sep 17 17:43:40.648254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:43:40.648224    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjxg6_800607ce-206f-4ce9-8edc-d6636d819654/serve-healthcheck-canary/0.log"
Sep 17 17:44:08.678245 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678209    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:44:08.678800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678567    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="alertmanager" containerID="cri-o://8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee" gracePeriod=120
Sep 17 17:44:08.678800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678629    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-metric" containerID="cri-o://5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd" gracePeriod=120
Sep 17 17:44:08.678800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678649    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy" containerID="cri-o://689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615" gracePeriod=120
Sep 17 17:44:08.678800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678671    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="config-reloader" containerID="cri-o://246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b" gracePeriod=120
Sep 17 17:44:08.678800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678697    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="prom-label-proxy" containerID="cri-o://3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de" gracePeriod=120
Sep 17 17:44:08.678800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:08.678637    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-web" containerID="cri-o://2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e" gracePeriod=120
Sep 17 17:44:09.796829 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796777    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de" exitCode=0
Sep 17 17:44:09.796829 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796828    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615" exitCode=0
Sep 17 17:44:09.797254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796838    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b" exitCode=0
Sep 17 17:44:09.797254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796847    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee" exitCode=0
Sep 17 17:44:09.797254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"}
Sep 17 17:44:09.797254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796946    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"}
Sep 17 17:44:09.797254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.796989    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"}
Sep 17 17:44:09.797254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.797030    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"}
Sep 17 17:44:09.930367 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.930347    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:09.996549 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996475    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-main-tls\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.996549 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996518    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-metrics-client-ca\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.996757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996553    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-cluster-tls-config\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.996757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996577    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-tls-assets\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.996757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996592    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-trusted-ca-bundle\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.996757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996631    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.996757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996655    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-main-db\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.997027 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.996988    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997019    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997039    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-volume\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997171    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997200    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kvp7\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-kube-api-access-2kvp7\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997247    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997255    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997277    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-out\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997302    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-web-config\") pod \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\" (UID: \"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb\") "
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997625    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-metrics-client-ca\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997646    2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:09.998314 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:09.997661    2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-alertmanager-main-db\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.000123 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.000086    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-volume" (OuterVolumeSpecName: "config-volume") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.000676 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.000622    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.001137 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.001108    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.001333 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.001314    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-kube-api-access-2kvp7" (OuterVolumeSpecName: "kube-api-access-2kvp7") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "kube-api-access-2kvp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:10.002492 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.002459    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.002851 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.002798    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.002996 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.002972    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:10.003102 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.003083    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-out" (OuterVolumeSpecName: "config-out") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:10.005547 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.005528    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.013406 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.013387    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-web-config" (OuterVolumeSpecName: "web-config") pod "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" (UID: "8305d83b-c6b5-4f04-a677-34b4dbe9f1fb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:10.098047 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098011    2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-cluster-tls-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098047 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098039    2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-tls-assets\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098047 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098050    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098061    2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-volume\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098070    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098079    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kvp7\" (UniqueName: \"kubernetes.io/projected/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-kube-api-access-2kvp7\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098090    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098098    2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-config-out\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098107    2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-web-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.098255 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.098114    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb-secret-alertmanager-main-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:10.808658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808628    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd" exitCode=0
Sep 17 17:44:10.808658 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808652    2585 generic.go:356] "Generic (PLEG): container finished" podID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerID="2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e" exitCode=0
Sep 17 17:44:10.809057 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808718    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:10.809057 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"}
Sep 17 17:44:10.809057 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808912    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"}
Sep 17 17:44:10.809057 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808957    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8305d83b-c6b5-4f04-a677-34b4dbe9f1fb","Type":"ContainerDied","Data":"ab347f8701eb43fb2acdc9fa9ea8b788af2028fd9e14c2ccd035e2f0f8b5ce01"}
Sep 17 17:44:10.809057 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.808972    2585 scope.go:117] "RemoveContainer" containerID="3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"
Sep 17 17:44:10.817343 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.817321    2585 scope.go:117] "RemoveContainer" containerID="5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"
Sep 17 17:44:10.824340 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.824323    2585 scope.go:117] "RemoveContainer" containerID="689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"
Sep 17 17:44:10.830901 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.830884    2585 scope.go:117] "RemoveContainer" containerID="2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"
Sep 17 17:44:10.837213 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.837199    2585 scope.go:117] "RemoveContainer" containerID="246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"
Sep 17 17:44:10.845661 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.845639    2585 scope.go:117] "RemoveContainer" containerID="8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"
Sep 17 17:44:10.847121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.847101    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:44:10.852566 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.852550    2585 scope.go:117] "RemoveContainer" containerID="8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"
Sep 17 17:44:10.859230 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.859214    2585 scope.go:117] "RemoveContainer" containerID="3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"
Sep 17 17:44:10.859512 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.859484    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de\": container with ID starting with 3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de not found: ID does not exist" containerID="3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"
Sep 17 17:44:10.859582 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.859524    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"} err="failed to get container status \"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de\": rpc error: code = NotFound desc = could not find container \"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de\": container with ID starting with 3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de not found: ID does not exist"
Sep 17 17:44:10.859582 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.859566    2585 scope.go:117] "RemoveContainer" containerID="5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"
Sep 17 17:44:10.859884 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.859865    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd\": container with ID starting with 5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd not found: ID does not exist" containerID="5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"
Sep 17 17:44:10.859973 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.859887    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"} err="failed to get container status \"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd\": rpc error: code = NotFound desc = could not find container \"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd\": container with ID starting with 5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd not found: ID does not exist"
Sep 17 17:44:10.859973 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.859901    2585 scope.go:117] "RemoveContainer" containerID="689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"
Sep 17 17:44:10.860159 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.860140    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615\": container with ID starting with 689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615 not found: ID does not exist" containerID="689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"
Sep 17 17:44:10.860223 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860168    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"} err="failed to get container status \"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615\": rpc error: code = NotFound desc = could not find container \"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615\": container with ID starting with 689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615 not found: ID does not exist"
Sep 17 17:44:10.860223 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860191    2585 scope.go:117] "RemoveContainer" containerID="2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"
Sep 17 17:44:10.860456 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.860430    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e\": container with ID starting with 2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e not found: ID does not exist" containerID="2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"
Sep 17 17:44:10.860492 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860460    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"} err="failed to get container status \"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e\": rpc error: code = NotFound desc = could not find container \"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e\": container with ID starting with 2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e not found: ID does not exist"
Sep 17 17:44:10.860492 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860475    2585 scope.go:117] "RemoveContainer" containerID="246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"
Sep 17 17:44:10.860689 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.860673    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b\": container with ID starting with 246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b not found: ID does not exist" containerID="246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"
Sep 17 17:44:10.860734 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860692    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"} err="failed to get container status \"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b\": rpc error: code = NotFound desc = could not find container \"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b\": container with ID starting with 246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b not found: ID does not exist"
Sep 17 17:44:10.860734 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860706    2585 scope.go:117] "RemoveContainer" containerID="8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"
Sep 17 17:44:10.860963 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.860945    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee\": container with ID starting with 8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee not found: ID does not exist" containerID="8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"
Sep 17 17:44:10.861033 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860969    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"} err="failed to get container status \"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee\": rpc error: code = NotFound desc = could not find container \"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee\": container with ID starting with 8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee not found: ID does not exist"
Sep 17 17:44:10.861033 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.860988    2585 scope.go:117] "RemoveContainer" containerID="8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"
Sep 17 17:44:10.861243 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:10.861228    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b\": container with ID starting with 8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b not found: ID does not exist" containerID="8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"
Sep 17 17:44:10.861277 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861247    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"} err="failed to get container status \"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b\": rpc error: code = NotFound desc = could not find container \"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b\": container with ID starting with 8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b not found: ID does not exist"
Sep 17 17:44:10.861277 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861260    2585 scope.go:117] "RemoveContainer" containerID="3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"
Sep 17 17:44:10.861473 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861455    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de"} err="failed to get container status \"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de\": rpc error: code = NotFound desc = could not find container \"3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de\": container with ID starting with 3812ed69d825511734a6bb31df3d05c21a672c92b3753f465b1f1b5bb0a5c7de not found: ID does not exist"
Sep 17 17:44:10.861538 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861473    2585 scope.go:117] "RemoveContainer" containerID="5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"
Sep 17 17:44:10.861695 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861673    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd"} err="failed to get container status \"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd\": rpc error: code = NotFound desc = could not find container \"5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd\": container with ID starting with 5a2194067e7a0ae2b8c044363df51941224a5fb430712c32d3692b8c3e164bcd not found: ID does not exist"
Sep 17 17:44:10.861781 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861697    2585 scope.go:117] "RemoveContainer" containerID="689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"
Sep 17 17:44:10.861962 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861944    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615"} err="failed to get container status \"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615\": rpc error: code = NotFound desc = could not find container \"689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615\": container with ID starting with 689da619337378a8e7a8f4a515812888f0c8c4cf65b581fdc2f9708129e59615 not found: ID does not exist"
Sep 17 17:44:10.862010 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.861963    2585 scope.go:117] "RemoveContainer" containerID="2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"
Sep 17 17:44:10.862187 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862170    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e"} err="failed to get container status \"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e\": rpc error: code = NotFound desc = could not find container \"2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e\": container with ID starting with 2421eebf5043ce15e9e05988b20d8bddeb68e22b0f6aa90290a48cd96631982e not found: ID does not exist"
Sep 17 17:44:10.862236 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862189    2585 scope.go:117] "RemoveContainer" containerID="246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"
Sep 17 17:44:10.862403 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862388    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b"} err="failed to get container status \"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b\": rpc error: code = NotFound desc = could not find container \"246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b\": container with ID starting with 246781430f9ba1d9a7e6da4dc5df1015182708ae9f467329147992708525603b not found: ID does not exist"
Sep 17 17:44:10.862454 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862403    2585 scope.go:117] "RemoveContainer" containerID="8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"
Sep 17 17:44:10.862630 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862609    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee"} err="failed to get container status \"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee\": rpc error: code = NotFound desc = could not find container \"8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee\": container with ID starting with 8b45afcfa76bcd6ad87f5cd59be872e9b9c6c68de67945809b9f2e3e3928e3ee not found: ID does not exist"
Sep 17 17:44:10.862673 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862632    2585 scope.go:117] "RemoveContainer" containerID="8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"
Sep 17 17:44:10.862845 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.862805    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b"} err="failed to get container status \"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b\": rpc error: code = NotFound desc = could not find container \"8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b\": container with ID starting with 8102080e79f834c1c1e9930c1748324155187ba98f62047e6e955594bec71c1b not found: ID does not exist"
Sep 17 17:44:10.875664 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.875641    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:44:10.915717 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.915694    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:44:10.915989 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.915978    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="config-reloader"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.915990    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="config-reloader"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916000    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-web"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916007    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-web"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916015    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="prom-label-proxy"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916020    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="prom-label-proxy"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916027    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="alertmanager"
Sep 17 17:44:10.916037 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916032    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="alertmanager"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916041    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916045    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916053    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="init-config-reloader"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916058    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="init-config-reloader"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916064    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-metric"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916069    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-metric"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916155    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="config-reloader"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916163    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916170    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-metric"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916175    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="prom-label-proxy"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916182    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="kube-rbac-proxy-web"
Sep 17 17:44:10.916231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.916189    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" containerName="alertmanager"
Sep 17 17:44:10.919165 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.919151    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:10.921984 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.921959    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:44:10.922071 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.921992    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ssm5f\""
Sep 17 17:44:10.922071 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922004    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:44:10.922184 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922007    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:44:10.922184 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922008    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:44:10.922284 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922189    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:44:10.922284 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922245    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:44:10.922383 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922352    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:44:10.922600 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.922587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:44:10.927795 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.927777    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:44:10.940253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.940232    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:44:10.958750 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:10.958724    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8305d83b-c6b5-4f04-a677-34b4dbe9f1fb" path="/var/lib/kubelet/pods/8305d83b-c6b5-4f04-a677-34b4dbe9f1fb/volumes"
Sep 17 17:44:11.007284 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007223    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e61d9c50-e55f-4b8d-9f94-035b088e918a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007284 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007253    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-web-config\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007284 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007324    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007356    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e61d9c50-e55f-4b8d-9f94-035b088e918a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-config-volume\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007421    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e61d9c50-e55f-4b8d-9f94-035b088e918a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007451    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007526    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e61d9c50-e55f-4b8d-9f94-035b088e918a-config-out\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007551    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e61d9c50-e55f-4b8d-9f94-035b088e918a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007855 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007585    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.007855 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.007632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntdr\" (UniqueName: \"kubernetes.io/projected/e61d9c50-e55f-4b8d-9f94-035b088e918a-kube-api-access-hntdr\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.107997 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.107959    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.107997 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108021    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e61d9c50-e55f-4b8d-9f94-035b088e918a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108040    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-config-volume\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108059    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e61d9c50-e55f-4b8d-9f94-035b088e918a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108087    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108108    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108128    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e61d9c50-e55f-4b8d-9f94-035b088e918a-config-out\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e61d9c50-e55f-4b8d-9f94-035b088e918a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108199 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108570 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108211    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntdr\" (UniqueName: \"kubernetes.io/projected/e61d9c50-e55f-4b8d-9f94-035b088e918a-kube-api-access-hntdr\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108570 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108248    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e61d9c50-e55f-4b8d-9f94-035b088e918a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.108570 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-web-config\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.109202 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108843    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e61d9c50-e55f-4b8d-9f94-035b088e918a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.109202 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.108892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e61d9c50-e55f-4b8d-9f94-035b088e918a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.110207 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.110140    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e61d9c50-e55f-4b8d-9f94-035b088e918a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.111681 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.111543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.111780 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.111738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-web-config\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.111780 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.111749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.112236 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.112214    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e61d9c50-e55f-4b8d-9f94-035b088e918a-config-out\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.112329 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.112224    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e61d9c50-e55f-4b8d-9f94-035b088e918a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.112390 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.112361    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.112390 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.112373    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.112574 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.112554    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.113301 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.113283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e61d9c50-e55f-4b8d-9f94-035b088e918a-config-volume\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.117378 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.117358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntdr\" (UniqueName: \"kubernetes.io/projected/e61d9c50-e55f-4b8d-9f94-035b088e918a-kube-api-access-hntdr\") pod \"alertmanager-main-0\" (UID: \"e61d9c50-e55f-4b8d-9f94-035b088e918a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.229278 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.229228    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:44:11.512942 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.512913    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:44:11.515690 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:44:11.515654    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61d9c50_e55f_4b8d_9f94_035b088e918a.slice/crio-7228c6a82eeb6fe3d6a32a843424ead8e01a15037300d61a2d32480d2425ab55 WatchSource:0}: Error finding container 7228c6a82eeb6fe3d6a32a843424ead8e01a15037300d61a2d32480d2425ab55: Status 404 returned error can't find the container with id 7228c6a82eeb6fe3d6a32a843424ead8e01a15037300d61a2d32480d2425ab55
Sep 17 17:44:11.814138 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.814059    2585 generic.go:356] "Generic (PLEG): container finished" podID="e61d9c50-e55f-4b8d-9f94-035b088e918a" containerID="1cb9b0842cf2317f28745a1eb01f7b342455e65c59f0a562d09bdaf886cb0523" exitCode=0
Sep 17 17:44:11.814512 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.814154    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerDied","Data":"1cb9b0842cf2317f28745a1eb01f7b342455e65c59f0a562d09bdaf886cb0523"}
Sep 17 17:44:11.814512 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:11.814234    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"7228c6a82eeb6fe3d6a32a843424ead8e01a15037300d61a2d32480d2425ab55"}
Sep 17 17:44:12.735311 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.735271    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"]
Sep 17 17:44:12.737715 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.737692    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.740362 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.740340    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:44:12.740476 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.740426    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:44:12.740655 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.740633    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:44:12.740712 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.740694    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vk5m5\""
Sep 17 17:44:12.740712 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.740698    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:44:12.740824 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.740797    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:44:12.745342 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.745318    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:44:12.750482 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.750462    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"]
Sep 17 17:44:12.821264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-serving-certs-ca-bundle\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-federate-client-tls\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821350    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-metrics-client-ca\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821372    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqh5v\" (UniqueName: \"kubernetes.io/projected/85521a94-1d73-4718-8b4b-a71da71cb442-kube-api-access-gqh5v\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821404    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-secret-telemeter-client\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821426    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-telemeter-client-tls\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821446    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.821716 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.821471    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.823071 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.823045    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"6883a9b4d2a1f95903d273dcbbdd8d30c921f5c1c88caceb20ffaf54aebbee74"}
Sep 17 17:44:12.823171 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.823124    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"6bb4422c909e2cc77a32ebad2545003edfd88b1ba3bfb856cdbf7edf7d69d01f"}
Sep 17 17:44:12.823230 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.823170    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"481251e3369b5b3d522a898af4f557add1df1953217d15dbe65db67b0e103701"}
Sep 17 17:44:12.823289 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.823222    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"04a329731e81820ab28a3a917ba8181a5be22f95b1ffcbec3a95c4178cfa0c57"}
Sep 17 17:44:12.823289 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.823266    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"85813be9d552b1c96d6c8e39a48b6eddb26649cc3e077d6936b9cee50ecfdc2a"}
Sep 17 17:44:12.823373 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.823306    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61d9c50-e55f-4b8d-9f94-035b088e918a","Type":"ContainerStarted","Data":"e810b0b6d24277afdbcf4a4e9f8403a3d8dceeaaf0693cff764a9803dcce747d"}
Sep 17 17:44:12.852664 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.852620    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8348067710000002 podStartE2EDuration="2.852605116s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.798345ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:12.847064407 +0000 UTC m=+186.038402106" watchObservedRunningTime="2026-09-17 17:44:12.852605116 +0000 UTC m=+186.043942800"
Sep 17 17:44:12.922348 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922303    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-serving-certs-ca-bundle\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.922512 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-federate-client-tls\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.922512 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922497    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-metrics-client-ca\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923065 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqh5v\" (UniqueName: \"kubernetes.io/projected/85521a94-1d73-4718-8b4b-a71da71cb442-kube-api-access-gqh5v\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923065 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-secret-telemeter-client\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923065 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-telemeter-client-tls\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923065 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923065 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.922893    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923591 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.923149    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-serving-certs-ca-bundle\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923591 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.923341    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-metrics-client-ca\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.923883 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.923861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85521a94-1d73-4718-8b4b-a71da71cb442-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.925352 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.925324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-telemeter-client-tls\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.925470 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.925448    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-federate-client-tls\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.925626 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.925609    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.925674 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.925659    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85521a94-1d73-4718-8b4b-a71da71cb442-secret-telemeter-client\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:12.934987 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:12.934967    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqh5v\" (UniqueName: \"kubernetes.io/projected/85521a94-1d73-4718-8b4b-a71da71cb442-kube-api-access-gqh5v\") pod \"telemeter-client-55d9d47dcb-7jkns\" (UID: \"85521a94-1d73-4718-8b4b-a71da71cb442\") " pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:13.052386 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:13.052313    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"
Sep 17 17:44:13.328176 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:13.328103    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55d9d47dcb-7jkns"]
Sep 17 17:44:13.331532 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:44:13.331501    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85521a94_1d73_4718_8b4b_a71da71cb442.slice/crio-ae562e0f0d2e11b54e6cce9dd25933047e835cbb396416715b2328f27bf1db81 WatchSource:0}: Error finding container ae562e0f0d2e11b54e6cce9dd25933047e835cbb396416715b2328f27bf1db81: Status 404 returned error can't find the container with id ae562e0f0d2e11b54e6cce9dd25933047e835cbb396416715b2328f27bf1db81
Sep 17 17:44:13.829411 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:13.829376    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns" event={"ID":"85521a94-1d73-4718-8b4b-a71da71cb442","Type":"ContainerStarted","Data":"ae562e0f0d2e11b54e6cce9dd25933047e835cbb396416715b2328f27bf1db81"}
Sep 17 17:44:15.840779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:15.840742    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns" event={"ID":"85521a94-1d73-4718-8b4b-a71da71cb442","Type":"ContainerStarted","Data":"6394742078be6804a75caecc94f2ed93ed8b10f08979ead7ab2f0234aac56e00"}
Sep 17 17:44:15.841151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:15.840845    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns" event={"ID":"85521a94-1d73-4718-8b4b-a71da71cb442","Type":"ContainerStarted","Data":"4749401af0f6cf0c86f1163fa4da94ce27a3ebd407f46863a1a18acbcb6f2af8"}
Sep 17 17:44:15.841151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:15.840865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns" event={"ID":"85521a94-1d73-4718-8b4b-a71da71cb442","Type":"ContainerStarted","Data":"94f07863def222568375f4af71c0ba09af97f65154d78b18ea75e90efe7a51ba"}
Sep 17 17:44:15.863264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:15.863203    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55d9d47dcb-7jkns" podStartSLOduration=2.066815227 podStartE2EDuration="3.863184538s" totalImagesPullingTime="1.796369311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:15.861835891 +0000 UTC m=+189.053173574" watchObservedRunningTime="2026-09-17 17:44:15.863184538 +0000 UTC m=+189.054522224"
Sep 17 17:44:26.478497 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:26.478467    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb47b8c79-wfwlp"]
Sep 17 17:44:51.498238 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.498199    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bb47b8c79-wfwlp" podUID="999a6c2e-bed4-4c07-8163-d36b2037955b" containerName="console" containerID="cri-o://692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f" gracePeriod=15
Sep 17 17:44:51.735257 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.735230    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:44:51.816290 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816221    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-oauth-serving-cert\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816290 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816260    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-serving-cert\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816293    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-console-config\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816334    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-oauth-config\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816369    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppxqq\" (UniqueName: \"kubernetes.io/projected/999a6c2e-bed4-4c07-8163-d36b2037955b-kube-api-access-ppxqq\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816449 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-trusted-ca-bundle\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816634 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816462    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-service-ca\") pod \"999a6c2e-bed4-4c07-8163-d36b2037955b\" (UID: \"999a6c2e-bed4-4c07-8163-d36b2037955b\") "
Sep 17 17:44:51.816745 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816722    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:51.816956 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816803    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-console-config" (OuterVolumeSpecName: "console-config") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:51.817026 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.816962    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:51.817067 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.817033    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-service-ca" (OuterVolumeSpecName: "service-ca") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:51.818792 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.818769    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:51.818792 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.818773    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:51.818909 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.818785    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999a6c2e-bed4-4c07-8163-d36b2037955b-kube-api-access-ppxqq" (OuterVolumeSpecName: "kube-api-access-ppxqq") pod "999a6c2e-bed4-4c07-8163-d36b2037955b" (UID: "999a6c2e-bed4-4c07-8163-d36b2037955b"). InnerVolumeSpecName "kube-api-access-ppxqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:51.917433 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917408    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppxqq\" (UniqueName: \"kubernetes.io/projected/999a6c2e-bed4-4c07-8163-d36b2037955b-kube-api-access-ppxqq\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.917433 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917431    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-trusted-ca-bundle\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.917564 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917440    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-service-ca\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.917564 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917449    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-oauth-serving-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.917564 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917458    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-serving-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.917564 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917467    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/999a6c2e-bed4-4c07-8163-d36b2037955b-console-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.917564 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.917475    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/999a6c2e-bed4-4c07-8163-d36b2037955b-console-oauth-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.988577 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.988547    2585 generic.go:356] "Generic (PLEG): container finished" podID="999a6c2e-bed4-4c07-8163-d36b2037955b" containerID="692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f" exitCode=0
Sep 17 17:44:51.988667 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.988584    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb47b8c79-wfwlp"
Sep 17 17:44:51.988667 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.988594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb47b8c79-wfwlp" event={"ID":"999a6c2e-bed4-4c07-8163-d36b2037955b","Type":"ContainerDied","Data":"692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f"}
Sep 17 17:44:51.988667 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.988635    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb47b8c79-wfwlp" event={"ID":"999a6c2e-bed4-4c07-8163-d36b2037955b","Type":"ContainerDied","Data":"c2cc53b1f69d95e1a7df0cd56584ab712f6f831e8e88fa8c4df009537e33ce35"}
Sep 17 17:44:51.988667 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.988661    2585 scope.go:117] "RemoveContainer" containerID="692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f"
Sep 17 17:44:51.997177 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.997157    2585 scope.go:117] "RemoveContainer" containerID="692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f"
Sep 17 17:44:51.997429 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:44:51.997411    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f\": container with ID starting with 692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f not found: ID does not exist" containerID="692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f"
Sep 17 17:44:51.997479 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:51.997435    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f"} err="failed to get container status \"692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f\": rpc error: code = NotFound desc = could not find container \"692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f\": container with ID starting with 692967a5cccba762b956d605f2087ca7a2c74bccf9c7c9510e0447a23eeb152f not found: ID does not exist"
Sep 17 17:44:52.009993 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:52.009972    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb47b8c79-wfwlp"]
Sep 17 17:44:52.014418 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:52.014390    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb47b8c79-wfwlp"]
Sep 17 17:44:52.959094 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:44:52.959067    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="999a6c2e-bed4-4c07-8163-d36b2037955b" path="/var/lib/kubelet/pods/999a6c2e-bed4-4c07-8163-d36b2037955b/volumes"
Sep 17 17:46:08.872703 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:08.872670    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:46:08.876633 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:08.876614    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:46:08.880293 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:08.880269    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:46:24.697703 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.697670    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"]
Sep 17 17:46:24.700197 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.697995    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="999a6c2e-bed4-4c07-8163-d36b2037955b" containerName="console"
Sep 17 17:46:24.700197 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.698007    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="999a6c2e-bed4-4c07-8163-d36b2037955b" containerName="console"
Sep 17 17:46:24.700197 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.698083    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="999a6c2e-bed4-4c07-8163-d36b2037955b" containerName="console"
Sep 17 17:46:24.701123 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.701108    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.704032 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.704010    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6csmg\""
Sep 17 17:46:24.704140 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.704051    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:46:24.704929 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.704913    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:46:24.716312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.716292    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"]
Sep 17 17:46:24.789426 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.789400    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jgj\" (UniqueName: \"kubernetes.io/projected/79c13103-863f-4ad0-b2b5-196a5300c345-kube-api-access-p4jgj\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.789561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.789436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.789561 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.789521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.890530 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.890507    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.890625 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.890584    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.890664 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.890619    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jgj\" (UniqueName: \"kubernetes.io/projected/79c13103-863f-4ad0-b2b5-196a5300c345-kube-api-access-p4jgj\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.890953 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.890934    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.890999 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.890941    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:24.901338 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:24.901312    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jgj\" (UniqueName: \"kubernetes.io/projected/79c13103-863f-4ad0-b2b5-196a5300c345-kube-api-access-p4jgj\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:25.009750 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:25.009679    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:25.287940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:25.287911    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"]
Sep 17 17:46:25.288793 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:46:25.288768    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c13103_863f_4ad0_b2b5_196a5300c345.slice/crio-8c3ca4ce5f76b2b14bffa933393e5122745c006403bd58e3e9160f9181ce02bf WatchSource:0}: Error finding container 8c3ca4ce5f76b2b14bffa933393e5122745c006403bd58e3e9160f9181ce02bf: Status 404 returned error can't find the container with id 8c3ca4ce5f76b2b14bffa933393e5122745c006403bd58e3e9160f9181ce02bf
Sep 17 17:46:25.290756 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:25.290741    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:46:25.349419 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:25.349391    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w" event={"ID":"79c13103-863f-4ad0-b2b5-196a5300c345","Type":"ContainerStarted","Data":"8c3ca4ce5f76b2b14bffa933393e5122745c006403bd58e3e9160f9181ce02bf"}
Sep 17 17:46:32.381592 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:32.381554    2585 generic.go:356] "Generic (PLEG): container finished" podID="79c13103-863f-4ad0-b2b5-196a5300c345" containerID="bbed49f4893b8c3c524086fef5dcffbb3a12cf61072ac95d5efcb164e73ada12" exitCode=0
Sep 17 17:46:32.381940 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:32.381599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w" event={"ID":"79c13103-863f-4ad0-b2b5-196a5300c345","Type":"ContainerDied","Data":"bbed49f4893b8c3c524086fef5dcffbb3a12cf61072ac95d5efcb164e73ada12"}
Sep 17 17:46:35.396491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:35.396458    2585 generic.go:356] "Generic (PLEG): container finished" podID="79c13103-863f-4ad0-b2b5-196a5300c345" containerID="66a169f1706e235efb2da6acfbed3e642f1f43188a9bafcdc80c90e18a16b429" exitCode=0
Sep 17 17:46:35.396963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:35.396506    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w" event={"ID":"79c13103-863f-4ad0-b2b5-196a5300c345","Type":"ContainerDied","Data":"66a169f1706e235efb2da6acfbed3e642f1f43188a9bafcdc80c90e18a16b429"}
Sep 17 17:46:43.433434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:43.433396    2585 generic.go:356] "Generic (PLEG): container finished" podID="79c13103-863f-4ad0-b2b5-196a5300c345" containerID="74589c4b9e55b7d499f569df467ff350d20447ec032a23f4517ddcf10f0d0395" exitCode=0
Sep 17 17:46:43.433892 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:43.433444    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w" event={"ID":"79c13103-863f-4ad0-b2b5-196a5300c345","Type":"ContainerDied","Data":"74589c4b9e55b7d499f569df467ff350d20447ec032a23f4517ddcf10f0d0395"}
Sep 17 17:46:44.556421 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.556397    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:44.651650 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.651622    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4jgj\" (UniqueName: \"kubernetes.io/projected/79c13103-863f-4ad0-b2b5-196a5300c345-kube-api-access-p4jgj\") pod \"79c13103-863f-4ad0-b2b5-196a5300c345\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") "
Sep 17 17:46:44.651796 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.651694    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-util\") pod \"79c13103-863f-4ad0-b2b5-196a5300c345\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") "
Sep 17 17:46:44.651796 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.651746    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-bundle\") pod \"79c13103-863f-4ad0-b2b5-196a5300c345\" (UID: \"79c13103-863f-4ad0-b2b5-196a5300c345\") "
Sep 17 17:46:44.652337 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.652316    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-bundle" (OuterVolumeSpecName: "bundle") pod "79c13103-863f-4ad0-b2b5-196a5300c345" (UID: "79c13103-863f-4ad0-b2b5-196a5300c345"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:44.654148 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.654127    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c13103-863f-4ad0-b2b5-196a5300c345-kube-api-access-p4jgj" (OuterVolumeSpecName: "kube-api-access-p4jgj") pod "79c13103-863f-4ad0-b2b5-196a5300c345" (UID: "79c13103-863f-4ad0-b2b5-196a5300c345"). InnerVolumeSpecName "kube-api-access-p4jgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:44.655670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.655651    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-util" (OuterVolumeSpecName: "util") pod "79c13103-863f-4ad0-b2b5-196a5300c345" (UID: "79c13103-863f-4ad0-b2b5-196a5300c345"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:44.752969 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.752899    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4jgj\" (UniqueName: \"kubernetes.io/projected/79c13103-863f-4ad0-b2b5-196a5300c345-kube-api-access-p4jgj\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:46:44.752969 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.752934    2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-util\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:46:44.752969 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:44.752945    2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79c13103-863f-4ad0-b2b5-196a5300c345-bundle\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.444392 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:45.444356    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w" event={"ID":"79c13103-863f-4ad0-b2b5-196a5300c345","Type":"ContainerDied","Data":"8c3ca4ce5f76b2b14bffa933393e5122745c006403bd58e3e9160f9181ce02bf"}
Sep 17 17:46:45.444555 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:45.444413    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3ca4ce5f76b2b14bffa933393e5122745c006403bd58e3e9160f9181ce02bf"
Sep 17 17:46:45.444555 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:45.444377    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anz62w"
Sep 17 17:46:53.203681 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.203607    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"]
Sep 17 17:46:53.204109 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204092    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="extract"
Sep 17 17:46:53.204154 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204111    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="extract"
Sep 17 17:46:53.204154 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204130    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="pull"
Sep 17 17:46:53.204154 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204135    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="pull"
Sep 17 17:46:53.204154 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204143    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="util"
Sep 17 17:46:53.204154 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204148    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="util"
Sep 17 17:46:53.204302 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.204206    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c13103-863f-4ad0-b2b5-196a5300c345" containerName="extract"
Sep 17 17:46:53.206957 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.206943    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.213151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.212888    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:46:53.213151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.213024    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:46:53.214804 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.214781    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:46:53.214934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.214827    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zzf5g\""
Sep 17 17:46:53.231503 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.231482    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"]
Sep 17 17:46:53.316084 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.316061    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d6c03cd3-2f31-4cfd-8792-2192910b3614-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-sd4pl\" (UID: \"d6c03cd3-2f31-4cfd-8792-2192910b3614\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.316209 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.316121    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4b2\" (UniqueName: \"kubernetes.io/projected/d6c03cd3-2f31-4cfd-8792-2192910b3614-kube-api-access-dw4b2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-sd4pl\" (UID: \"d6c03cd3-2f31-4cfd-8792-2192910b3614\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.417094 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.417065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d6c03cd3-2f31-4cfd-8792-2192910b3614-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-sd4pl\" (UID: \"d6c03cd3-2f31-4cfd-8792-2192910b3614\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.417226 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.417142    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4b2\" (UniqueName: \"kubernetes.io/projected/d6c03cd3-2f31-4cfd-8792-2192910b3614-kube-api-access-dw4b2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-sd4pl\" (UID: \"d6c03cd3-2f31-4cfd-8792-2192910b3614\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.419418 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.419400    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d6c03cd3-2f31-4cfd-8792-2192910b3614-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-sd4pl\" (UID: \"d6c03cd3-2f31-4cfd-8792-2192910b3614\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.428695 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.428663    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4b2\" (UniqueName: \"kubernetes.io/projected/d6c03cd3-2f31-4cfd-8792-2192910b3614-kube-api-access-dw4b2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-sd4pl\" (UID: \"d6c03cd3-2f31-4cfd-8792-2192910b3614\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.516732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.516664    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:53.805629 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:53.805554    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"]
Sep 17 17:46:53.808365 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:46:53.808339    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c03cd3_2f31_4cfd_8792_2192910b3614.slice/crio-58f407c235693acbf97f093c92142b9b9f68fc092511c46bc36b501f06f5c4ff WatchSource:0}: Error finding container 58f407c235693acbf97f093c92142b9b9f68fc092511c46bc36b501f06f5c4ff: Status 404 returned error can't find the container with id 58f407c235693acbf97f093c92142b9b9f68fc092511c46bc36b501f06f5c4ff
Sep 17 17:46:54.479642 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:54.479607    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl" event={"ID":"d6c03cd3-2f31-4cfd-8792-2192910b3614","Type":"ContainerStarted","Data":"58f407c235693acbf97f093c92142b9b9f68fc092511c46bc36b501f06f5c4ff"}
Sep 17 17:46:57.496547 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:57.496512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl" event={"ID":"d6c03cd3-2f31-4cfd-8792-2192910b3614","Type":"ContainerStarted","Data":"9d432e2a1924b12272f755598c526f78ca5bc18ffbe9a31f4b25062f616bab6a"}
Sep 17 17:46:57.496958 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:57.496663    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:46:58.801123 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.801067    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl" podStartSLOduration=2.642784859 podStartE2EDuration="5.801049134s" totalImagesPullingTime="3.158264275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:57.544849011 +0000 UTC m=+350.736186695" watchObservedRunningTime="2026-09-17 17:46:58.801049134 +0000 UTC m=+351.992386856"
Sep 17 17:46:58.802510 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.802486    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dsgn7"]
Sep 17 17:46:58.810621 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.810577    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:58.813931 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.813911    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:46:58.815178 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.815148    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-crk65\""
Sep 17 17:46:58.815293 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.815193    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:46:58.828077 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.828057    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dsgn7"]
Sep 17 17:46:58.864223 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.864197    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8w5\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-kube-api-access-vv8w5\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:58.864329 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.864240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-certificates\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:58.964739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.964701    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-certificates\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:58.964868 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.964783    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8w5\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-kube-api-access-vv8w5\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:58.964868 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:46:58.964849    2585 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 17:46:58.964868 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:46:58.964868    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-dsgn7: secret "keda-admission-webhooks-certs" not found
Sep 17 17:46:58.964975 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:46:58.964926    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-certificates podName:de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:59.46491112 +0000 UTC m=+352.656248783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-certificates") pod "keda-admission-6c4d864b86-dsgn7" (UID: "de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58") : secret "keda-admission-webhooks-certs" not found
Sep 17 17:46:58.989523 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:58.989497    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8w5\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-kube-api-access-vv8w5\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:59.468850 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:59.468801    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-certificates\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:59.471500 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:59.471467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58-certificates\") pod \"keda-admission-6c4d864b86-dsgn7\" (UID: \"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58\") " pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:59.721279 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:59.721196    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:46:59.991068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:46:59.991040    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dsgn7"]
Sep 17 17:46:59.992921 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:46:59.992890    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde72f3b4_a5d7_48dc_a1f3_2f30ef85fb58.slice/crio-2d9ba2caca7fd2a8fa5635113eaf56b5c155d557a10bd93254dbea59122e9e1f WatchSource:0}: Error finding container 2d9ba2caca7fd2a8fa5635113eaf56b5c155d557a10bd93254dbea59122e9e1f: Status 404 returned error can't find the container with id 2d9ba2caca7fd2a8fa5635113eaf56b5c155d557a10bd93254dbea59122e9e1f
Sep 17 17:47:00.511487 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:47:00.511454    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7" event={"ID":"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58","Type":"ContainerStarted","Data":"2d9ba2caca7fd2a8fa5635113eaf56b5c155d557a10bd93254dbea59122e9e1f"}
Sep 17 17:47:08.545909 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:47:08.545871    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7" event={"ID":"de72f3b4-a5d7-48dc-a1f3-2f30ef85fb58","Type":"ContainerStarted","Data":"651ea6601434e48158f7edade3b2fd82187f99f0625dcdeec40535cfc1cbb05d"}
Sep 17 17:47:08.546278 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:47:08.545931    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:47:08.563060 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:47:08.563015    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7" podStartSLOduration=2.205276021 podStartE2EDuration="10.563002078s" totalImagesPullingTime="8.357726057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:08.561922867 +0000 UTC m=+361.753260554" watchObservedRunningTime="2026-09-17 17:47:08.563002078 +0000 UTC m=+361.754339762"
Sep 17 17:47:18.503910 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:47:18.503880    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-sd4pl"
Sep 17 17:47:29.552769 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:47:29.552741    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-dsgn7"
Sep 17 17:48:19.242593 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.242505    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-t4cqt"]
Sep 17 17:48:19.246068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.246046    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.248885 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.248865    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:48:19.248986 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.248881    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-kpf7m\""
Sep 17 17:48:19.249916 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.249895    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:48:19.250296 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.250278    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:48:19.257563 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.257541    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-t4cqt"]
Sep 17 17:48:19.278652 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.278628    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-4p625"]
Sep 17 17:48:19.282358 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.282342    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.286542 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.286526    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dcrjv\""
Sep 17 17:48:19.286650 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.286561    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:48:19.296139 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.296117    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4p625"]
Sep 17 17:48:19.373469 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.373439    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc22d\" (UniqueName: \"kubernetes.io/projected/4334a790-ec3b-42e2-8c64-b6831ea3aa5f-kube-api-access-tc22d\") pod \"seaweedfs-64568bcd49-4p625\" (UID: \"4334a790-ec3b-42e2-8c64-b6831ea3aa5f\") " pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.373596 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.373484    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4334a790-ec3b-42e2-8c64-b6831ea3aa5f-data\") pod \"seaweedfs-64568bcd49-4p625\" (UID: \"4334a790-ec3b-42e2-8c64-b6831ea3aa5f\") " pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.373596 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.373506    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65drl\" (UniqueName: \"kubernetes.io/projected/91949a59-6265-477f-a860-70d767b8f444-kube-api-access-65drl\") pod \"kserve-controller-manager-5c5cdf4877-t4cqt\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") " pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.373670 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.373617    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91949a59-6265-477f-a860-70d767b8f444-cert\") pod \"kserve-controller-manager-5c5cdf4877-t4cqt\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") " pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.474459 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.474427    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65drl\" (UniqueName: \"kubernetes.io/projected/91949a59-6265-477f-a860-70d767b8f444-kube-api-access-65drl\") pod \"kserve-controller-manager-5c5cdf4877-t4cqt\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") " pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.474569 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.474478    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91949a59-6265-477f-a860-70d767b8f444-cert\") pod \"kserve-controller-manager-5c5cdf4877-t4cqt\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") " pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.474569 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.474533    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc22d\" (UniqueName: \"kubernetes.io/projected/4334a790-ec3b-42e2-8c64-b6831ea3aa5f-kube-api-access-tc22d\") pod \"seaweedfs-64568bcd49-4p625\" (UID: \"4334a790-ec3b-42e2-8c64-b6831ea3aa5f\") " pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.474569 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.474563    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4334a790-ec3b-42e2-8c64-b6831ea3aa5f-data\") pod \"seaweedfs-64568bcd49-4p625\" (UID: \"4334a790-ec3b-42e2-8c64-b6831ea3aa5f\") " pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.475001 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.474982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4334a790-ec3b-42e2-8c64-b6831ea3aa5f-data\") pod \"seaweedfs-64568bcd49-4p625\" (UID: \"4334a790-ec3b-42e2-8c64-b6831ea3aa5f\") " pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.477143 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.477111    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91949a59-6265-477f-a860-70d767b8f444-cert\") pod \"kserve-controller-manager-5c5cdf4877-t4cqt\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") " pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.485829 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.484538    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65drl\" (UniqueName: \"kubernetes.io/projected/91949a59-6265-477f-a860-70d767b8f444-kube-api-access-65drl\") pod \"kserve-controller-manager-5c5cdf4877-t4cqt\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") " pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.488086 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.487337    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc22d\" (UniqueName: \"kubernetes.io/projected/4334a790-ec3b-42e2-8c64-b6831ea3aa5f-kube-api-access-tc22d\") pod \"seaweedfs-64568bcd49-4p625\" (UID: \"4334a790-ec3b-42e2-8c64-b6831ea3aa5f\") " pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.557381 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.557310    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:19.591554 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.591523    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:19.827776 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.827750    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-t4cqt"]
Sep 17 17:48:19.829557 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:48:19.829534    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91949a59_6265_477f_a860_70d767b8f444.slice/crio-2dc552ea0f066ce46b9716fa2ab4a7f7de22c1c2b6dfe63e3c1bce715d6d65e9 WatchSource:0}: Error finding container 2dc552ea0f066ce46b9716fa2ab4a7f7de22c1c2b6dfe63e3c1bce715d6d65e9: Status 404 returned error can't find the container with id 2dc552ea0f066ce46b9716fa2ab4a7f7de22c1c2b6dfe63e3c1bce715d6d65e9
Sep 17 17:48:19.866006 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.865977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt" event={"ID":"91949a59-6265-477f-a860-70d767b8f444","Type":"ContainerStarted","Data":"2dc552ea0f066ce46b9716fa2ab4a7f7de22c1c2b6dfe63e3c1bce715d6d65e9"}
Sep 17 17:48:19.872514 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:19.872493    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4p625"]
Sep 17 17:48:19.872943 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:48:19.872921    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4334a790_ec3b_42e2_8c64_b6831ea3aa5f.slice/crio-9d96ef41f7fdfeb031d8eeb2f1980585c1f41b2d2f0adb6f132834c63008b92e WatchSource:0}: Error finding container 9d96ef41f7fdfeb031d8eeb2f1980585c1f41b2d2f0adb6f132834c63008b92e: Status 404 returned error can't find the container with id 9d96ef41f7fdfeb031d8eeb2f1980585c1f41b2d2f0adb6f132834c63008b92e
Sep 17 17:48:20.874759 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:20.874724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4p625" event={"ID":"4334a790-ec3b-42e2-8c64-b6831ea3aa5f","Type":"ContainerStarted","Data":"9d96ef41f7fdfeb031d8eeb2f1980585c1f41b2d2f0adb6f132834c63008b92e"}
Sep 17 17:48:23.891110 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:23.891069    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt" event={"ID":"91949a59-6265-477f-a860-70d767b8f444","Type":"ContainerStarted","Data":"19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a"}
Sep 17 17:48:23.891541 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:23.891136    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:48:23.892721 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:23.892699    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4p625" event={"ID":"4334a790-ec3b-42e2-8c64-b6831ea3aa5f","Type":"ContainerStarted","Data":"95e10e1a74690ca36fef07fbb0d2416f0de385915478d13818fbee470076a107"}
Sep 17 17:48:23.892868 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:23.892848    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:23.918403 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:23.918352    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt" podStartSLOduration=1.326624987 podStartE2EDuration="4.918339076s" totalImagesPullingTime="3.591714089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:23.917338658 +0000 UTC m=+437.108676347" watchObservedRunningTime="2026-09-17 17:48:23.918339076 +0000 UTC m=+437.109676761"
Sep 17 17:48:23.938262 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:23.938224    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-4p625" podStartSLOduration=1.303423065 podStartE2EDuration="4.938213065s" totalImagesPullingTime="3.63479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:23.937589633 +0000 UTC m=+437.128927318" watchObservedRunningTime="2026-09-17 17:48:23.938213065 +0000 UTC m=+437.129550749"
Sep 17 17:48:29.899646 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:29.899614    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-4p625"
Sep 17 17:48:54.904312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:48:54.904281    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:49:03.346120 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.346086    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-t4cqt"]
Sep 17 17:49:03.346523 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.346289    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt" podUID="91949a59-6265-477f-a860-70d767b8f444" containerName="manager" containerID="cri-o://19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a" gracePeriod=10
Sep 17 17:49:03.371433 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.371409    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-4qgtw"]
Sep 17 17:49:03.377171 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.375163    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.387020 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.387001    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-4qgtw"]
Sep 17 17:49:03.403957 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.403932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/688356f1-2037-4d78-bd3d-93e037a91c57-cert\") pod \"kserve-controller-manager-5c5cdf4877-4qgtw\" (UID: \"688356f1-2037-4d78-bd3d-93e037a91c57\") " pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.404067 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.404002    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56p42\" (UniqueName: \"kubernetes.io/projected/688356f1-2037-4d78-bd3d-93e037a91c57-kube-api-access-56p42\") pod \"kserve-controller-manager-5c5cdf4877-4qgtw\" (UID: \"688356f1-2037-4d78-bd3d-93e037a91c57\") " pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.504962 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.504929    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56p42\" (UniqueName: \"kubernetes.io/projected/688356f1-2037-4d78-bd3d-93e037a91c57-kube-api-access-56p42\") pod \"kserve-controller-manager-5c5cdf4877-4qgtw\" (UID: \"688356f1-2037-4d78-bd3d-93e037a91c57\") " pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.505126 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.505001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/688356f1-2037-4d78-bd3d-93e037a91c57-cert\") pod \"kserve-controller-manager-5c5cdf4877-4qgtw\" (UID: \"688356f1-2037-4d78-bd3d-93e037a91c57\") " pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.507508 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.507481    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/688356f1-2037-4d78-bd3d-93e037a91c57-cert\") pod \"kserve-controller-manager-5c5cdf4877-4qgtw\" (UID: \"688356f1-2037-4d78-bd3d-93e037a91c57\") " pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.516305 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.516281    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56p42\" (UniqueName: \"kubernetes.io/projected/688356f1-2037-4d78-bd3d-93e037a91c57-kube-api-access-56p42\") pod \"kserve-controller-manager-5c5cdf4877-4qgtw\" (UID: \"688356f1-2037-4d78-bd3d-93e037a91c57\") " pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:03.579869 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.579843    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:49:03.605850 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.605771    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65drl\" (UniqueName: \"kubernetes.io/projected/91949a59-6265-477f-a860-70d767b8f444-kube-api-access-65drl\") pod \"91949a59-6265-477f-a860-70d767b8f444\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") "
Sep 17 17:49:03.605850 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.605834    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91949a59-6265-477f-a860-70d767b8f444-cert\") pod \"91949a59-6265-477f-a860-70d767b8f444\" (UID: \"91949a59-6265-477f-a860-70d767b8f444\") "
Sep 17 17:49:03.608366 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.608333    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91949a59-6265-477f-a860-70d767b8f444-cert" (OuterVolumeSpecName: "cert") pod "91949a59-6265-477f-a860-70d767b8f444" (UID: "91949a59-6265-477f-a860-70d767b8f444"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:03.608366 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.608353    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91949a59-6265-477f-a860-70d767b8f444-kube-api-access-65drl" (OuterVolumeSpecName: "kube-api-access-65drl") pod "91949a59-6265-477f-a860-70d767b8f444" (UID: "91949a59-6265-477f-a860-70d767b8f444"). InnerVolumeSpecName "kube-api-access-65drl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:03.706739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.706709    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65drl\" (UniqueName: \"kubernetes.io/projected/91949a59-6265-477f-a860-70d767b8f444-kube-api-access-65drl\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:49:03.706739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.706733    2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91949a59-6265-477f-a860-70d767b8f444-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:49:03.732661 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:03.732641    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:04.005198 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.005175    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-4qgtw"]
Sep 17 17:49:04.007081 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:49:04.007062    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688356f1_2037_4d78_bd3d_93e037a91c57.slice/crio-5b051b9f0ca95135edbac1d8d20d92332bd37b15960323992351b88e8ba939a1 WatchSource:0}: Error finding container 5b051b9f0ca95135edbac1d8d20d92332bd37b15960323992351b88e8ba939a1: Status 404 returned error can't find the container with id 5b051b9f0ca95135edbac1d8d20d92332bd37b15960323992351b88e8ba939a1
Sep 17 17:49:04.074470 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.074439    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw" event={"ID":"688356f1-2037-4d78-bd3d-93e037a91c57","Type":"ContainerStarted","Data":"5b051b9f0ca95135edbac1d8d20d92332bd37b15960323992351b88e8ba939a1"}
Sep 17 17:49:04.075572 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.075549    2585 generic.go:356] "Generic (PLEG): container finished" podID="91949a59-6265-477f-a860-70d767b8f444" containerID="19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a" exitCode=0
Sep 17 17:49:04.075651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.075594    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt"
Sep 17 17:49:04.075651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.075635    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt" event={"ID":"91949a59-6265-477f-a860-70d767b8f444","Type":"ContainerDied","Data":"19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a"}
Sep 17 17:49:04.075739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.075684    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-t4cqt" event={"ID":"91949a59-6265-477f-a860-70d767b8f444","Type":"ContainerDied","Data":"2dc552ea0f066ce46b9716fa2ab4a7f7de22c1c2b6dfe63e3c1bce715d6d65e9"}
Sep 17 17:49:04.075739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.075707    2585 scope.go:117] "RemoveContainer" containerID="19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a"
Sep 17 17:49:04.084295 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.084263    2585 scope.go:117] "RemoveContainer" containerID="19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a"
Sep 17 17:49:04.084561 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:49:04.084544    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a\": container with ID starting with 19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a not found: ID does not exist" containerID="19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a"
Sep 17 17:49:04.084613 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.084570    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a"} err="failed to get container status \"19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a\": rpc error: code = NotFound desc = could not find container \"19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a\": container with ID starting with 19251870e715274dd9f389439d287ac7a7970ddac9ad4d7db518ee231061cf9a not found: ID does not exist"
Sep 17 17:49:04.097264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.097239    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-t4cqt"]
Sep 17 17:49:04.101431 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.101411    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-t4cqt"]
Sep 17 17:49:04.960226 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:04.960197    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91949a59-6265-477f-a860-70d767b8f444" path="/var/lib/kubelet/pods/91949a59-6265-477f-a860-70d767b8f444/volumes"
Sep 17 17:49:05.081100 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:05.081072    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw" event={"ID":"688356f1-2037-4d78-bd3d-93e037a91c57","Type":"ContainerStarted","Data":"570d3883e58172fa9bac20474fd3fe9e1b0ab1280faa2a89874c53abd550c823"}
Sep 17 17:49:05.081261 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:05.081176    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:49:05.102294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:05.102243    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw" podStartSLOduration=1.735529322 podStartE2EDuration="2.102228107s" totalImagesPullingTime="366.698785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:05.101244117 +0000 UTC m=+478.292581801" watchObservedRunningTime="2026-09-17 17:49:05.102228107 +0000 UTC m=+478.293565791"
Sep 17 17:49:36.091845 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:49:36.091801    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-4qgtw"
Sep 17 17:51:08.904060 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:51:08.904029    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:51:08.904967 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:51:08.904948    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:52:02.814518 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.814480    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"]
Sep 17 17:52:02.815024 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.814997    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91949a59-6265-477f-a860-70d767b8f444" containerName="manager"
Sep 17 17:52:02.815024 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.815013    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="91949a59-6265-477f-a860-70d767b8f444" containerName="manager"
Sep 17 17:52:02.815130 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.815119    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="91949a59-6265-477f-a860-70d767b8f444" containerName="manager"
Sep 17 17:52:02.818448 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.818431    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.821162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.821140    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 17:52:02.821162 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.821150    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 17:52:02.829784 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.829758    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"]
Sep 17 17:52:02.886273 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.886243    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.886376 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.886309    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ac9556ce-5089-4792-9983-79819b5824a0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.886376 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.886350    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zfl\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-kube-api-access-49zfl\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.987099 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.987077    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.987198 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.987119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ac9556ce-5089-4792-9983-79819b5824a0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.987198 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.987139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zfl\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-kube-api-access-49zfl\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.987290 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:52:02.987216    2585 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 17 17:52:02.987290 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:52:02.987232    2585 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-qf62g: secret "seaweedfs-tls-serving" not found
Sep 17 17:52:02.987290 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:52:02.987285    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-seaweedfs-tls-serving podName:ac9556ce-5089-4792-9983-79819b5824a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:52:03.487269274 +0000 UTC m=+656.678606955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-qf62g" (UID: "ac9556ce-5089-4792-9983-79819b5824a0") : secret "seaweedfs-tls-serving" not found
Sep 17 17:52:02.987518 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.987499    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ac9556ce-5089-4792-9983-79819b5824a0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:02.997254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:02.997237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zfl\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-kube-api-access-49zfl\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:03.490719 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:03.490666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:03.493124 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:03.493103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ac9556ce-5089-4792-9983-79819b5824a0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-qf62g\" (UID: \"ac9556ce-5089-4792-9983-79819b5824a0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:03.727662 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:03.727634    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"
Sep 17 17:52:03.996303 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:03.996279    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-qf62g"]
Sep 17 17:52:03.998376 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:52:03.998349    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9556ce_5089_4792_9983_79819b5824a0.slice/crio-579fc3ce1615992ac66a7865e5a97f16961761f90fb2387ed5c6e35b13607418 WatchSource:0}: Error finding container 579fc3ce1615992ac66a7865e5a97f16961761f90fb2387ed5c6e35b13607418: Status 404 returned error can't find the container with id 579fc3ce1615992ac66a7865e5a97f16961761f90fb2387ed5c6e35b13607418
Sep 17 17:52:03.999632 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:03.999616    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:52:04.877970 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:04.877936    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g" event={"ID":"ac9556ce-5089-4792-9983-79819b5824a0","Type":"ContainerStarted","Data":"9af0e239ae5be21188ca16e6c11e96dedfd10b9424864a13ffb68b81a9a02563"}
Sep 17 17:52:04.878130 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:04.877980    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g" event={"ID":"ac9556ce-5089-4792-9983-79819b5824a0","Type":"ContainerStarted","Data":"579fc3ce1615992ac66a7865e5a97f16961761f90fb2387ed5c6e35b13607418"}
Sep 17 17:52:04.911426 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:04.911380    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-qf62g" podStartSLOduration=2.602076664 podStartE2EDuration="2.911363949s" totalImagesPullingTime="309.287285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:04.908884094 +0000 UTC m=+658.100221779" watchObservedRunningTime="2026-09-17 17:52:04.911363949 +0000 UTC m=+658.102701654"
Sep 17 17:52:30.237382 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.237343    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"]
Sep 17 17:52:30.241735 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.241702    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.244444 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.244405    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 17:52:30.244609 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.244444    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:52:30.244720 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.244657    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:52:30.245006 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.244981    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 17:52:30.245006 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.245004    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:52:30.245176 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.245004    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 17:52:30.245596 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.245578    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:52:30.251711 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.251688    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"]
Sep 17 17:52:30.399203 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.399176    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.399375 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.399214    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4362db6-ac9f-4678-ab1a-31f51129728b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.399375 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.399249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4362db6-ac9f-4678-ab1a-31f51129728b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.399375 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.399325    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hs5\" (UniqueName: \"kubernetes.io/projected/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-api-access-56hs5\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.399521 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.399375    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.500426 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.500347    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.500426 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.500395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.500426 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.500418    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4362db6-ac9f-4678-ab1a-31f51129728b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.500695 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.500437    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4362db6-ac9f-4678-ab1a-31f51129728b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.500695 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.500482    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hs5\" (UniqueName: \"kubernetes.io/projected/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-api-access-56hs5\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.500896 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.500869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4362db6-ac9f-4678-ab1a-31f51129728b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.501163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.501142    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.501269 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.501166    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.503082 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.503065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4362db6-ac9f-4678-ab1a-31f51129728b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.509046 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.509024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hs5\" (UniqueName: \"kubernetes.io/projected/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-api-access-56hs5\") pod \"isvc-sklearn-batcher-predictor-56c7449565-pdj8q\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.554963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.554934    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:30.834016 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:30.833993    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"]
Sep 17 17:52:30.835914 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:52:30.835882    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4362db6_ac9f_4678_ab1a_31f51129728b.slice/crio-2c678d228663996a2a7d2435c907d99248b5f3e1ce0f7c04138b2c6f2852e883 WatchSource:0}: Error finding container 2c678d228663996a2a7d2435c907d99248b5f3e1ce0f7c04138b2c6f2852e883: Status 404 returned error can't find the container with id 2c678d228663996a2a7d2435c907d99248b5f3e1ce0f7c04138b2c6f2852e883
Sep 17 17:52:31.003041 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:31.003010    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerStarted","Data":"2c678d228663996a2a7d2435c907d99248b5f3e1ce0f7c04138b2c6f2852e883"}
Sep 17 17:52:34.019962 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:34.019873    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerStarted","Data":"71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f"}
Sep 17 17:52:38.040013 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:38.039977    2585 generic.go:356] "Generic (PLEG): container finished" podID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerID="71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f" exitCode=0
Sep 17 17:52:38.040370 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:38.040049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerDied","Data":"71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f"}
Sep 17 17:52:52.129192 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:52.129162    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerStarted","Data":"c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79"}
Sep 17 17:52:55.147531 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:55.147498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerStarted","Data":"c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903"}
Sep 17 17:52:57.158434 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:57.158383    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerStarted","Data":"fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3"}
Sep 17 17:52:57.158928 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:57.158520    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:57.184325 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:57.184265    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podStartSLOduration=1.840158864 podStartE2EDuration="27.184244693s" totalImagesPullingTime="21.574835217s" totalInitContainerRuntime="3.769250612s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:30 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:57.18269905 +0000 UTC m=+710.374036737" watchObservedRunningTime="2026-09-17 17:52:57.184244693 +0000 UTC m=+710.375582379"
Sep 17 17:52:58.163848 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:58.163796    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:58.164282 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:58.163893    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:52:58.165073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:58.165044    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:52:58.165706 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:58.165681    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:59.168597 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:59.168551    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:52:59.169071 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:59.168883    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:59.172219 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:52:59.172198    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:53:00.173154 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:00.173117    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:53:00.173519 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:00.173410    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:10.173785 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:10.173739    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:53:10.174291 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:10.174271    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:20.173960 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:20.173913    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:53:20.174437 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:20.174413    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:30.173555 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:30.173512    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:53:30.174044 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:30.174021    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:40.173096 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:40.173052    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:53:40.173595 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:40.173572    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:50.173983 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:50.173932    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:53:50.174480 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:53:50.174375    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:00.173508 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:00.173460    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:54:00.174069 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:00.173999    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:10.173993 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:10.173963    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:10.174441 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:10.174052    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:25.114397 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.114364    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"]
Sep 17 17:54:25.114914 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.114686    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" containerID="cri-o://c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79" gracePeriod=30
Sep 17 17:54:25.114914 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.114744    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" containerID="cri-o://fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3" gracePeriod=30
Sep 17 17:54:25.114914 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.114738    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" containerID="cri-o://c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903" gracePeriod=30
Sep 17 17:54:25.176375 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.176345    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"]
Sep 17 17:54:25.180375 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.180357    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.183123 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.183103    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 17:54:25.183226 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.183110    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 17:54:25.192503 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.192483    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"]
Sep 17 17:54:25.335912 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.335882    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.336050 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.335916    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37bc1d9e-2062-4be5-8663-1044f4ef39ad-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.336050 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.335973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdjp\" (UniqueName: \"kubernetes.io/projected/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-api-access-shdjp\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.336050 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.336024    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.336050 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.336048    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.437176 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdjp\" (UniqueName: \"kubernetes.io/projected/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-api-access-shdjp\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.437301 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437215    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.437301 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.437369 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437328    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.437369 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437347    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37bc1d9e-2062-4be5-8663-1044f4ef39ad-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.437600 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437580    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.438020 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.437990    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.438020 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.438019    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.439871 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.439849    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37bc1d9e-2062-4be5-8663-1044f4ef39ad-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.446457 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.446434    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdjp\" (UniqueName: \"kubernetes.io/projected/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-api-access-shdjp\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.491267 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.491237    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:25.568132 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.568095    2585 generic.go:356] "Generic (PLEG): container finished" podID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerID="c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903" exitCode=2
Sep 17 17:54:25.568264 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.568162    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerDied","Data":"c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903"}
Sep 17 17:54:25.762541 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:25.762513    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"]
Sep 17 17:54:25.763841 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:54:25.763798    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bc1d9e_2062_4be5_8663_1044f4ef39ad.slice/crio-086fd35479cc494b7a622afe661ea8e2e70a7533526369755a3528bc206dd026 WatchSource:0}: Error finding container 086fd35479cc494b7a622afe661ea8e2e70a7533526369755a3528bc206dd026: Status 404 returned error can't find the container with id 086fd35479cc494b7a622afe661ea8e2e70a7533526369755a3528bc206dd026
Sep 17 17:54:26.574819 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:26.574773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerStarted","Data":"ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587"}
Sep 17 17:54:26.575164 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:26.574878    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerStarted","Data":"086fd35479cc494b7a622afe661ea8e2e70a7533526369755a3528bc206dd026"}
Sep 17 17:54:29.169400 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:29.169362    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:54:29.591072 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:29.591040    2585 generic.go:356] "Generic (PLEG): container finished" podID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerID="c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79" exitCode=0
Sep 17 17:54:29.591240 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:29.591109    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerDied","Data":"c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79"}
Sep 17 17:54:29.592720 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:29.592694    2585 generic.go:356] "Generic (PLEG): container finished" podID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerID="ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587" exitCode=0
Sep 17 17:54:29.592829 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:29.592752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerDied","Data":"ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587"}
Sep 17 17:54:30.173820 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.173779    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:54:30.174223 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.174157    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:30.600432 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.600359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerStarted","Data":"b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d"}
Sep 17 17:54:30.600432 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.600425    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerStarted","Data":"ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8"}
Sep 17 17:54:30.600629 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.600451    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerStarted","Data":"efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5"}
Sep 17 17:54:30.600757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.600734    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:30.600848 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.600828    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:30.600898 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.600865    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:30.602169 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.602139    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:54:30.602800 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.602776    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:30.623521 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:30.623481    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podStartSLOduration=1.996487312 podStartE2EDuration="5.623469413s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626982101s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:30.622151895 +0000 UTC m=+803.813489579" watchObservedRunningTime="2026-09-17 17:54:30.623469413 +0000 UTC m=+803.814807098"
Sep 17 17:54:31.606573 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:31.606522    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:54:31.607089 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:31.606826    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:34.169295 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:34.169258    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:54:36.610367 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:36.610332    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:54:36.610904 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:36.610879    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:54:36.611196 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:36.611174    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:39.168850 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:39.168789    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:54:39.169211 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:39.168947    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:40.173458 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:40.173419    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:54:40.173901 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:40.173737    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:44.169521 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:44.169470    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:54:46.611294 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:46.611231    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:54:46.611763 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:46.611701    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:49.169408 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:49.169366    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:54:50.173143 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:50.173100    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:54:50.173616 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:50.173226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:50.173616 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:50.173403    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:50.173616 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:50.173494    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:54.169323 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:54.169281    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:54:55.269220 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.269195    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:55.386121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56hs5\" (UniqueName: \"kubernetes.io/projected/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-api-access-56hs5\") pod \"f4362db6-ac9f-4678-ab1a-31f51129728b\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") "
Sep 17 17:54:55.386121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386086    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4362db6-ac9f-4678-ab1a-31f51129728b-proxy-tls\") pod \"f4362db6-ac9f-4678-ab1a-31f51129728b\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") "
Sep 17 17:54:55.386121 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386115    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-cabundle-cert\") pod \"f4362db6-ac9f-4678-ab1a-31f51129728b\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") "
Sep 17 17:54:55.386374 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386135    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-rbac-proxy-sar-config\") pod \"f4362db6-ac9f-4678-ab1a-31f51129728b\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") "
Sep 17 17:54:55.386374 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386172    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4362db6-ac9f-4678-ab1a-31f51129728b-kserve-provision-location\") pod \"f4362db6-ac9f-4678-ab1a-31f51129728b\" (UID: \"f4362db6-ac9f-4678-ab1a-31f51129728b\") "
Sep 17 17:54:55.386490 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386460    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4362db6-ac9f-4678-ab1a-31f51129728b" (UID: "f4362db6-ac9f-4678-ab1a-31f51129728b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:55.386549 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386498    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4362db6-ac9f-4678-ab1a-31f51129728b" (UID: "f4362db6-ac9f-4678-ab1a-31f51129728b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:55.386675 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.386650    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4362db6-ac9f-4678-ab1a-31f51129728b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4362db6-ac9f-4678-ab1a-31f51129728b" (UID: "f4362db6-ac9f-4678-ab1a-31f51129728b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:55.388499 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.388475    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-api-access-56hs5" (OuterVolumeSpecName: "kube-api-access-56hs5") pod "f4362db6-ac9f-4678-ab1a-31f51129728b" (UID: "f4362db6-ac9f-4678-ab1a-31f51129728b"). InnerVolumeSpecName "kube-api-access-56hs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:55.388499 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.388491    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4362db6-ac9f-4678-ab1a-31f51129728b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4362db6-ac9f-4678-ab1a-31f51129728b" (UID: "f4362db6-ac9f-4678-ab1a-31f51129728b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:55.487287 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.487253    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:54:55.487287 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.487282    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4362db6-ac9f-4678-ab1a-31f51129728b-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:54:55.487287 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.487292    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56hs5\" (UniqueName: \"kubernetes.io/projected/f4362db6-ac9f-4678-ab1a-31f51129728b-kube-api-access-56hs5\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:54:55.487488 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.487302    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4362db6-ac9f-4678-ab1a-31f51129728b-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:54:55.487488 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.487311    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4362db6-ac9f-4678-ab1a-31f51129728b-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:54:55.723179 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.723096    2585 generic.go:356] "Generic (PLEG): container finished" podID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerID="fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3" exitCode=0
Sep 17 17:54:55.723179 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.723147    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerDied","Data":"fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3"}
Sep 17 17:54:55.723179 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.723170    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"
Sep 17 17:54:55.723372 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.723230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q" event={"ID":"f4362db6-ac9f-4678-ab1a-31f51129728b","Type":"ContainerDied","Data":"2c678d228663996a2a7d2435c907d99248b5f3e1ce0f7c04138b2c6f2852e883"}
Sep 17 17:54:55.723372 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.723262    2585 scope.go:117] "RemoveContainer" containerID="fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3"
Sep 17 17:54:55.732244 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.732228    2585 scope.go:117] "RemoveContainer" containerID="c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903"
Sep 17 17:54:55.740230 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.740213    2585 scope.go:117] "RemoveContainer" containerID="c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79"
Sep 17 17:54:55.747224 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.747200    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"]
Sep 17 17:54:55.747832 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.747800    2585 scope.go:117] "RemoveContainer" containerID="71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f"
Sep 17 17:54:55.754275 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.754254    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-pdj8q"]
Sep 17 17:54:55.756243 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.756224    2585 scope.go:117] "RemoveContainer" containerID="fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3"
Sep 17 17:54:55.756514 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:54:55.756492    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3\": container with ID starting with fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3 not found: ID does not exist" containerID="fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3"
Sep 17 17:54:55.756601 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.756521    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3"} err="failed to get container status \"fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3\": rpc error: code = NotFound desc = could not find container \"fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3\": container with ID starting with fac883dc924379e139e6e9e5ef60d178dee5ea2dbb2fc5228f45722aea8e15e3 not found: ID does not exist"
Sep 17 17:54:55.756601 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.756538    2585 scope.go:117] "RemoveContainer" containerID="c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903"
Sep 17 17:54:55.756866 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:54:55.756848    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903\": container with ID starting with c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903 not found: ID does not exist" containerID="c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903"
Sep 17 17:54:55.756939 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.756877    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903"} err="failed to get container status \"c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903\": rpc error: code = NotFound desc = could not find container \"c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903\": container with ID starting with c7c472ef5ec5fa160fa19534e527abbf8d2ebb70c78ced5c25758df521aae903 not found: ID does not exist"
Sep 17 17:54:55.756939 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.756899    2585 scope.go:117] "RemoveContainer" containerID="c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79"
Sep 17 17:54:55.757160 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:54:55.757142    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79\": container with ID starting with c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79 not found: ID does not exist" containerID="c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79"
Sep 17 17:54:55.757224 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.757167    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79"} err="failed to get container status \"c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79\": rpc error: code = NotFound desc = could not find container \"c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79\": container with ID starting with c23c3c216916f6e7249f56f7765da5ae898b4e56406fe91b72fa108192913d79 not found: ID does not exist"
Sep 17 17:54:55.757224 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.757185    2585 scope.go:117] "RemoveContainer" containerID="71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f"
Sep 17 17:54:55.757473 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:54:55.757456    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f\": container with ID starting with 71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f not found: ID does not exist" containerID="71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f"
Sep 17 17:54:55.757510 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:55.757478    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f"} err="failed to get container status \"71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f\": rpc error: code = NotFound desc = could not find container \"71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f\": container with ID starting with 71c6c4734444b18f48dce37909a0e775583bf6150440c8389b66ada932777b8f not found: ID does not exist"
Sep 17 17:54:56.611631 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:56.611591    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:54:56.612103 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:56.612080    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:56.960365 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:54:56.960333    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" path="/var/lib/kubelet/pods/f4362db6-ac9f-4678-ab1a-31f51129728b/volumes"
Sep 17 17:55:06.611491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:06.611449    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:55:06.612084 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:06.611856    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:16.611161 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:16.611107    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:55:16.611549 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:16.611491    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:26.611367 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:26.611321    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:55:26.611846 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:26.611758    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:36.611415 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:36.611378    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:55:36.611946 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:36.611920    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:46.611677 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:46.611649    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:55:46.612155 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:55:46.611738    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:56:00.246382 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.246344    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"]
Sep 17 17:56:00.246788 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.246624    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" containerID="cri-o://efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5" gracePeriod=30
Sep 17 17:56:00.246788 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.246651    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" containerID="cri-o://b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d" gracePeriod=30
Sep 17 17:56:00.246788 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.246699    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" containerID="cri-o://ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8" gracePeriod=30
Sep 17 17:56:00.359673 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.359644    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"]
Sep 17 17:56:00.360052 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360030    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="storage-initializer"
Sep 17 17:56:00.360052 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360046    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="storage-initializer"
Sep 17 17:56:00.360144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360073    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy"
Sep 17 17:56:00.360144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360083    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy"
Sep 17 17:56:00.360144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360097    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent"
Sep 17 17:56:00.360144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360102    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent"
Sep 17 17:56:00.360144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360109    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container"
Sep 17 17:56:00.360144 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360115    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container"
Sep 17 17:56:00.360345 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360229    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kserve-container"
Sep 17 17:56:00.360345 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360243    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="kube-rbac-proxy"
Sep 17 17:56:00.360345 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.360251    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4362db6-ac9f-4678-ab1a-31f51129728b" containerName="agent"
Sep 17 17:56:00.363296 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.363278    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.365760 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.365740    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 17:56:00.365949 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.365932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 17:56:00.375461 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.375423    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"]
Sep 17 17:56:00.419283 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.419253    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.419400 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.419312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.419400 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.419383    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72nww\" (UniqueName: \"kubernetes.io/projected/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-api-access-72nww\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.520732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.520645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.520732 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.520706    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.520964 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.520742    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72nww\" (UniqueName: \"kubernetes.io/projected/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-api-access-72nww\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.520964 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:56:00.520897    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 17:56:00.521072 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:56:00.520969    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls podName:8bf57785-d01b-46a1-8dcd-6e0195d0d4a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:01.020947963 +0000 UTC m=+894.212285626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-p8dvx" (UID: "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 17:56:00.521411 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.521388    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:00.529864 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:00.529843    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72nww\" (UniqueName: \"kubernetes.io/projected/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-api-access-72nww\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:01.021513 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.021479    2585 generic.go:356] "Generic (PLEG): container finished" podID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerID="ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8" exitCode=2
Sep 17 17:56:01.021666 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.021553    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerDied","Data":"ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8"}
Sep 17 17:56:01.024797 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.024778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:01.027241 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.027217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-p8dvx\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:01.274722 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.274626    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:01.551909 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.548573    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"]
Sep 17 17:56:01.606563 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:01.606531    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:56:02.027045 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:02.027010    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" event={"ID":"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5","Type":"ContainerStarted","Data":"68247b6d7c655ca8de9309c6956a2dc830511b2ee8e8ff010b1f976fdc1f3b52"}
Sep 17 17:56:03.033530 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:03.033446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" event={"ID":"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5","Type":"ContainerStarted","Data":"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"}
Sep 17 17:56:03.033530 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:03.033507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" event={"ID":"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5","Type":"ContainerStarted","Data":"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"}
Sep 17 17:56:03.034043 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:03.033540    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:03.034043 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:03.033598    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:03.035510 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:03.035492    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:03.055414 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:03.055373    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" podStartSLOduration=1.8336792530000001 podStartE2EDuration="3.055361283s" totalImagesPullingTime="1.22168203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:03.053975981 +0000 UTC m=+896.245313666" watchObservedRunningTime="2026-09-17 17:56:03.055361283 +0000 UTC m=+896.246698969"
Sep 17 17:56:05.045522 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:05.045491    2585 generic.go:356] "Generic (PLEG): container finished" podID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerID="efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5" exitCode=0
Sep 17 17:56:05.045943 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:05.045563    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerDied","Data":"efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5"}
Sep 17 17:56:06.607284 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:06.607246    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:56:06.611509 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:06.611488    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:56:06.611944 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:06.611919    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:08.940304 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:08.940277    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:56:08.940304 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:08.940306    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 17:56:10.050275 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.050248    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:56:10.397102 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.397070    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"]
Sep 17 17:56:10.400964 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.400944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.403479 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.403460    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 17:56:10.403691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.403668    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 17:56:10.413659 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.413635    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"]
Sep 17 17:56:10.504515 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.504489    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-cabundle-cert\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.504642 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.504524    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.504642 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.504542    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84638809-bcd2-4539-b828-5496cddb9a17-kserve-provision-location\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.504744 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.504656    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84638809-bcd2-4539-b828-5496cddb9a17-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.504790 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.504768    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccng7\" (UniqueName: \"kubernetes.io/projected/84638809-bcd2-4539-b828-5496cddb9a17-kube-api-access-ccng7\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.605787 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.605748    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84638809-bcd2-4539-b828-5496cddb9a17-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.605944 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.605859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccng7\" (UniqueName: \"kubernetes.io/projected/84638809-bcd2-4539-b828-5496cddb9a17-kube-api-access-ccng7\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.605944 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.605907    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-cabundle-cert\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.606072 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.605947    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.606072 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.605969    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84638809-bcd2-4539-b828-5496cddb9a17-kserve-provision-location\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.606424 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.606397    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84638809-bcd2-4539-b828-5496cddb9a17-kserve-provision-location\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.606731 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.606702    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.606865 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.606850    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-cabundle-cert\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.608556 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.608532    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84638809-bcd2-4539-b828-5496cddb9a17-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.614938 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.614896    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccng7\" (UniqueName: \"kubernetes.io/projected/84638809-bcd2-4539-b828-5496cddb9a17-kube-api-access-ccng7\") pod \"isvc-logger-predictor-664ccfb57d-wfstv\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.711901 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.711849    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:10.983666 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:10.983640    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"]
Sep 17 17:56:10.985573 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:56:10.985542    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84638809_bcd2_4539_b828_5496cddb9a17.slice/crio-290881d0f55098a97798392ffb254cb34419a96c7467292efa37f004c5f171e0 WatchSource:0}: Error finding container 290881d0f55098a97798392ffb254cb34419a96c7467292efa37f004c5f171e0: Status 404 returned error can't find the container with id 290881d0f55098a97798392ffb254cb34419a96c7467292efa37f004c5f171e0
Sep 17 17:56:11.073983 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:11.073946    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerStarted","Data":"c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97"}
Sep 17 17:56:11.074332 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:11.074023    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerStarted","Data":"290881d0f55098a97798392ffb254cb34419a96c7467292efa37f004c5f171e0"}
Sep 17 17:56:11.606582 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:11.606544    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:56:11.606757 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:11.606694    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:56:15.093061 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:15.093030    2585 generic.go:356] "Generic (PLEG): container finished" podID="84638809-bcd2-4539-b828-5496cddb9a17" containerID="c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97" exitCode=0
Sep 17 17:56:15.093396 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:15.093102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerDied","Data":"c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97"}
Sep 17 17:56:16.100965 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.100934    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerStarted","Data":"069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922"}
Sep 17 17:56:16.101603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.100996    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerStarted","Data":"ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb"}
Sep 17 17:56:16.101603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.101026    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerStarted","Data":"3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6"}
Sep 17 17:56:16.101603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.101329    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:16.101603 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.101460    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:16.103023 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.102994    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:16.125019 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.124977    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podStartSLOduration=2.276174931 podStartE2EDuration="6.124961451s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.84878652s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:16.123929277 +0000 UTC m=+909.315266963" watchObservedRunningTime="2026-09-17 17:56:16.124961451 +0000 UTC m=+909.316299136"
Sep 17 17:56:16.606996 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.606950    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:56:16.611220 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.611197    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:56:16.611547 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:16.611524    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:17.105389 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:17.105355    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:17.105887 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:17.105492    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:17.106796 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:17.106762    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:18.110182 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:18.110140    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:18.110725 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:18.110604    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:21.607245 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:21.607199    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:56:23.114195 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:23.114165    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:56:23.114956 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:23.114923    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:23.115120 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:23.115089    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:26.607290 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:26.607247    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:56:26.611454 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:26.611430    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 17 17:56:26.611555 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:26.611539    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:56:26.611876 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:26.611847    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:26.611985 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:26.611939    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:56:30.398553 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.398528    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:56:30.474527 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474502    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kserve-provision-location\") pod \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") "
Sep 17 17:56:30.474649 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474540    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-cabundle-cert\") pod \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") "
Sep 17 17:56:30.474649 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474567    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-rbac-proxy-sar-config\") pod \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") "
Sep 17 17:56:30.474649 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474636    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shdjp\" (UniqueName: \"kubernetes.io/projected/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-api-access-shdjp\") pod \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") "
Sep 17 17:56:30.474762 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474689    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37bc1d9e-2062-4be5-8663-1044f4ef39ad-proxy-tls\") pod \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\" (UID: \"37bc1d9e-2062-4be5-8663-1044f4ef39ad\") "
Sep 17 17:56:30.474887 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474866    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37bc1d9e-2062-4be5-8663-1044f4ef39ad" (UID: "37bc1d9e-2062-4be5-8663-1044f4ef39ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:30.474963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474914    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37bc1d9e-2062-4be5-8663-1044f4ef39ad" (UID: "37bc1d9e-2062-4be5-8663-1044f4ef39ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:30.474963 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474919    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37bc1d9e-2062-4be5-8663-1044f4ef39ad" (UID: "37bc1d9e-2062-4be5-8663-1044f4ef39ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:30.475073 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.474968    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:56:30.476797 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.476776    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37bc1d9e-2062-4be5-8663-1044f4ef39ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37bc1d9e-2062-4be5-8663-1044f4ef39ad" (UID: "37bc1d9e-2062-4be5-8663-1044f4ef39ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:30.477008 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.476990    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-api-access-shdjp" (OuterVolumeSpecName: "kube-api-access-shdjp") pod "37bc1d9e-2062-4be5-8663-1044f4ef39ad" (UID: "37bc1d9e-2062-4be5-8663-1044f4ef39ad"). InnerVolumeSpecName "kube-api-access-shdjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:30.575476 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.575399    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37bc1d9e-2062-4be5-8663-1044f4ef39ad-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:56:30.575476 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.575438    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:56:30.575476 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.575449    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:56:30.575476 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:30.575459    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shdjp\" (UniqueName: \"kubernetes.io/projected/37bc1d9e-2062-4be5-8663-1044f4ef39ad-kube-api-access-shdjp\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:56:31.172395 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.172358    2585 generic.go:356] "Generic (PLEG): container finished" podID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerID="b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d" exitCode=0
Sep 17 17:56:31.172528 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.172428    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"
Sep 17 17:56:31.172528 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.172448    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerDied","Data":"b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d"}
Sep 17 17:56:31.172528 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.172516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7" event={"ID":"37bc1d9e-2062-4be5-8663-1044f4ef39ad","Type":"ContainerDied","Data":"086fd35479cc494b7a622afe661ea8e2e70a7533526369755a3528bc206dd026"}
Sep 17 17:56:31.172650 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.172548    2585 scope.go:117] "RemoveContainer" containerID="b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d"
Sep 17 17:56:31.180917 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.180896    2585 scope.go:117] "RemoveContainer" containerID="ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8"
Sep 17 17:56:31.188293 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.188275    2585 scope.go:117] "RemoveContainer" containerID="efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5"
Sep 17 17:56:31.193926 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.193905    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"]
Sep 17 17:56:31.195994 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.195978    2585 scope.go:117] "RemoveContainer" containerID="ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587"
Sep 17 17:56:31.200736 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.200719    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-dsdp7"]
Sep 17 17:56:31.203830 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.203787    2585 scope.go:117] "RemoveContainer" containerID="b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d"
Sep 17 17:56:31.204100 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:56:31.204081    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d\": container with ID starting with b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d not found: ID does not exist" containerID="b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d"
Sep 17 17:56:31.204151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204110    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d"} err="failed to get container status \"b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d\": rpc error: code = NotFound desc = could not find container \"b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d\": container with ID starting with b6400a4be50eb1345b6898fdf7c9cf20f4bfe1b894f3a9de9372c5fc8beda28d not found: ID does not exist"
Sep 17 17:56:31.204151 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204127    2585 scope.go:117] "RemoveContainer" containerID="ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8"
Sep 17 17:56:31.204388 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:56:31.204370    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8\": container with ID starting with ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8 not found: ID does not exist" containerID="ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8"
Sep 17 17:56:31.204428 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204394    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8"} err="failed to get container status \"ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8\": rpc error: code = NotFound desc = could not find container \"ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8\": container with ID starting with ed82a0d6ccca09856b711f15c9431699ca8eb25f5f196b2543dc6869b76023a8 not found: ID does not exist"
Sep 17 17:56:31.204428 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204408    2585 scope.go:117] "RemoveContainer" containerID="efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5"
Sep 17 17:56:31.204651 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:56:31.204637    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5\": container with ID starting with efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5 not found: ID does not exist" containerID="efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5"
Sep 17 17:56:31.204691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204653    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5"} err="failed to get container status \"efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5\": rpc error: code = NotFound desc = could not find container \"efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5\": container with ID starting with efac0bb97e898e8642646f04173948d8ba5734dba9a5bb2ad65da8f283c2adb5 not found: ID does not exist"
Sep 17 17:56:31.204691 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204665    2585 scope.go:117] "RemoveContainer" containerID="ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587"
Sep 17 17:56:31.204891 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:56:31.204873    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587\": container with ID starting with ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587 not found: ID does not exist" containerID="ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587"
Sep 17 17:56:31.204960 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:31.204895    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587"} err="failed to get container status \"ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587\": rpc error: code = NotFound desc = could not find container \"ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587\": container with ID starting with ee307af78dcf458d845bc0cf9d2a7c76c0860b278f43aa6df5b6de13c28a3587 not found: ID does not exist"
Sep 17 17:56:32.959793 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:32.959759    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" path="/var/lib/kubelet/pods/37bc1d9e-2062-4be5-8663-1044f4ef39ad/volumes"
Sep 17 17:56:33.115063 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:33.115022    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:33.115504 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:33.115475    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:36.786977 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.786945    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"]
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787327    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787338    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787349    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787354    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787364    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="storage-initializer"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787369    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="storage-initializer"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787380    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy"
Sep 17 17:56:36.787401 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787386    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy"
Sep 17 17:56:36.787642 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787436    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="agent"
Sep 17 17:56:36.787642 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787445    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kube-rbac-proxy"
Sep 17 17:56:36.787642 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.787452    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="37bc1d9e-2062-4be5-8663-1044f4ef39ad" containerName="kserve-container"
Sep 17 17:56:36.794040 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.794017    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.797312 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.797290    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 17:56:36.797566 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.797553    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 17:56:36.809389 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.809371    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"]
Sep 17 17:56:36.826323 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.826301    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgdz\" (UniqueName: \"kubernetes.io/projected/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-api-access-8pgdz\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.826441 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.826329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.826441 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.826347    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.826565 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.826494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a81288-f843-4d87-ba2b-cdd5d14660fb-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.826565 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.826552    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.927323 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.927291    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a81288-f843-4d87-ba2b-cdd5d14660fb-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.927505 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.927333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.927505 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.927363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgdz\" (UniqueName: \"kubernetes.io/projected/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-api-access-8pgdz\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.927505 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.927385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.927505 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.927411    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.927773 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.927752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.928095 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.928076    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.928145 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.928093    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.930015 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.929994    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a81288-f843-4d87-ba2b-cdd5d14660fb-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:36.939324 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:36.939295    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgdz\" (UniqueName: \"kubernetes.io/projected/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-api-access-8pgdz\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:37.106984 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:37.106900    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:56:37.384565 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:37.384540    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"]
Sep 17 17:56:37.386099 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:56:37.386072    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a81288_f843_4d87_ba2b_cdd5d14660fb.slice/crio-f9ada4e8dfd2c50a09e180f6323e1496f0a03ee7949e90dd948211223e944dd1 WatchSource:0}: Error finding container f9ada4e8dfd2c50a09e180f6323e1496f0a03ee7949e90dd948211223e944dd1: Status 404 returned error can't find the container with id f9ada4e8dfd2c50a09e180f6323e1496f0a03ee7949e90dd948211223e944dd1
Sep 17 17:56:38.214886 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:38.214850    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerStarted","Data":"86523bf2f707495756ddbe1d3f1c04d9049194851e52e5a2e2d24f774b042709"}
Sep 17 17:56:38.215243 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:38.214918    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerStarted","Data":"f9ada4e8dfd2c50a09e180f6323e1496f0a03ee7949e90dd948211223e944dd1"}
Sep 17 17:56:42.235460 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:42.235422    2585 generic.go:356] "Generic (PLEG): container finished" podID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerID="86523bf2f707495756ddbe1d3f1c04d9049194851e52e5a2e2d24f774b042709" exitCode=0
Sep 17 17:56:42.235839 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:42.235471    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerDied","Data":"86523bf2f707495756ddbe1d3f1c04d9049194851e52e5a2e2d24f774b042709"}
Sep 17 17:56:43.115983 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:43.115038    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:43.117135 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:43.117079    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:53.114874 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:53.114769    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:56:53.138226 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:56:53.115368    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:03.115443 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:03.115407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:57:03.115969 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:03.115945    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:03.372332 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:03.372301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerStarted","Data":"5404dedac78c96ff01cbc373592406529c9f8e91525c8663fbdace2a9f629d4d"}
Sep 17 17:57:03.372475 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:03.372370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerStarted","Data":"a6840dcd91807f1b75c38f71ed0bebbcd4cb5fa77c6557b9c70e3b76d6dda356"}
Sep 17 17:57:03.372574 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:03.372562    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:57:03.398379 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:03.398316    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podStartSLOduration=2.771847049 podStartE2EDuration="27.3982952s" totalImagesPullingTime="20.840348605s" totalInitContainerRuntime="3.786099546s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:03.395129567 +0000 UTC m=+956.586467276" watchObservedRunningTime="2026-09-17 17:57:03.3982952 +0000 UTC m=+956.589632885"
Sep 17 17:57:04.378248 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:04.378220    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:57:04.379493 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:04.379465    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:05.382546 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:05.382507    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:10.386951 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:10.386916    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:57:10.387595 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:10.387563    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:13.115412 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:13.115367    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:57:13.115806 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:13.115742    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:20.387540 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:20.387498    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:23.115032 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:23.114988    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:57:23.115470 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:23.115447    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:30.387540 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:30.387495    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:32.960491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:32.960464    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:57:32.960873 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:32.960540    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:57:40.387524 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:40.387489    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:45.436521 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.436481    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-p8dvx_8bf57785-d01b-46a1-8dcd-6e0195d0d4a5/kserve-container/0.log"
Sep 17 17:57:45.815023 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.814946    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"]
Sep 17 17:57:45.815270 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.815242    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" containerID="cri-o://3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6" gracePeriod=30
Sep 17 17:57:45.815367 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.815260    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" containerID="cri-o://ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb" gracePeriod=30
Sep 17 17:57:45.815367 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.815242    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" containerID="cri-o://069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922" gracePeriod=30
Sep 17 17:57:45.959163 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.959129    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"]
Sep 17 17:57:45.959394 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.959367    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kserve-container" containerID="cri-o://c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989" gracePeriod=30
Sep 17 17:57:45.959464 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:45.959412    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kube-rbac-proxy" containerID="cri-o://3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec" gracePeriod=30
Sep 17 17:57:46.206693 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.206667    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:57:46.325107 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.325077    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls\") pod \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") "
Sep 17 17:57:46.325107 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.325117    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72nww\" (UniqueName: \"kubernetes.io/projected/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-api-access-72nww\") pod \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") "
Sep 17 17:57:46.325396 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.325149    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-rbac-proxy-sar-config\") pod \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\" (UID: \"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5\") "
Sep 17 17:57:46.325559 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.325529    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" (UID: "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:46.327386 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.327364    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-api-access-72nww" (OuterVolumeSpecName: "kube-api-access-72nww") pod "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" (UID: "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5"). InnerVolumeSpecName "kube-api-access-72nww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:46.327468 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.327411    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" (UID: "8bf57785-d01b-46a1-8dcd-6e0195d0d4a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:46.426651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.426617    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:57:46.426651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.426645    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72nww\" (UniqueName: \"kubernetes.io/projected/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-api-access-72nww\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:57:46.426651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.426657    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:57:46.580254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580173    2585 generic.go:356] "Generic (PLEG): container finished" podID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerID="3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec" exitCode=2
Sep 17 17:57:46.580254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580196    2585 generic.go:356] "Generic (PLEG): container finished" podID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerID="c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989" exitCode=2
Sep 17 17:57:46.580254 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580227    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"
Sep 17 17:57:46.580823 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580263    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" event={"ID":"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5","Type":"ContainerDied","Data":"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"}
Sep 17 17:57:46.580823 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580342    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" event={"ID":"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5","Type":"ContainerDied","Data":"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"}
Sep 17 17:57:46.580823 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580377    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx" event={"ID":"8bf57785-d01b-46a1-8dcd-6e0195d0d4a5","Type":"ContainerDied","Data":"68247b6d7c655ca8de9309c6956a2dc830511b2ee8e8ff010b1f976fdc1f3b52"}
Sep 17 17:57:46.580823 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.580381    2585 scope.go:117] "RemoveContainer" containerID="3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"
Sep 17 17:57:46.583145 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.583122    2585 generic.go:356] "Generic (PLEG): container finished" podID="84638809-bcd2-4539-b828-5496cddb9a17" containerID="ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb" exitCode=2
Sep 17 17:57:46.583265 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.583167    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerDied","Data":"ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb"}
Sep 17 17:57:46.589536 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.589517    2585 scope.go:117] "RemoveContainer" containerID="c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"
Sep 17 17:57:46.597181 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597164    2585 scope.go:117] "RemoveContainer" containerID="3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"
Sep 17 17:57:46.597436 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:57:46.597418    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec\": container with ID starting with 3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec not found: ID does not exist" containerID="3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"
Sep 17 17:57:46.597491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597442    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"} err="failed to get container status \"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec\": rpc error: code = NotFound desc = could not find container \"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec\": container with ID starting with 3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec not found: ID does not exist"
Sep 17 17:57:46.597491 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597458    2585 scope.go:117] "RemoveContainer" containerID="c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"
Sep 17 17:57:46.597681 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:57:46.597660    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989\": container with ID starting with c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989 not found: ID does not exist" containerID="c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"
Sep 17 17:57:46.597749 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597691    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"} err="failed to get container status \"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989\": rpc error: code = NotFound desc = could not find container \"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989\": container with ID starting with c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989 not found: ID does not exist"
Sep 17 17:57:46.597749 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597714    2585 scope.go:117] "RemoveContainer" containerID="3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"
Sep 17 17:57:46.597975 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597953    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec"} err="failed to get container status \"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec\": rpc error: code = NotFound desc = could not find container \"3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec\": container with ID starting with 3a341474a5b535ef1665274210061d6858df1976f2c5c8de250300217ea99cec not found: ID does not exist"
Sep 17 17:57:46.598032 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.597976    2585 scope.go:117] "RemoveContainer" containerID="c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"
Sep 17 17:57:46.598215 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.598197    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989"} err="failed to get container status \"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989\": rpc error: code = NotFound desc = could not find container \"c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989\": container with ID starting with c200972a3c56b581053f126ec4c5747dd3e7c23ed07df7c79b2b769ef3bd8989 not found: ID does not exist"
Sep 17 17:57:46.606306 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.606284    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"]
Sep 17 17:57:46.609699 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.609679    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-p8dvx"]
Sep 17 17:57:46.960299 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:46.960271    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" path="/var/lib/kubelet/pods/8bf57785-d01b-46a1-8dcd-6e0195d0d4a5/volumes"
Sep 17 17:57:48.111055 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:48.111011    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:57:50.387709 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:50.387671    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:57:50.607740 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:50.607712    2585 generic.go:356] "Generic (PLEG): container finished" podID="84638809-bcd2-4539-b828-5496cddb9a17" containerID="3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6" exitCode=0
Sep 17 17:57:50.607897 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:50.607788    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerDied","Data":"3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6"}
Sep 17 17:57:52.954936 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:52.954887    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:57:52.955352 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:52.955275    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:53.110625 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:53.110589    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:57:58.110712 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:58.110667    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:57:58.111111 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:57:58.110786    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:58:00.388090 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:00.388048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:58:02.955244 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:02.955191    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:02.955646 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:02.955557    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:03.111145 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:03.111109    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:58:08.111041 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:08.110999    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:58:10.388260 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:10.388221    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:58:12.954907 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:12.954856    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:12.955439 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:12.955121    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:12.959588 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:12.959564    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:58:12.959718 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:12.959658    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:58:13.110979 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:13.110942    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:58:15.969857 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:15.969830    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:58:16.064666 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.064635    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84638809-bcd2-4539-b828-5496cddb9a17-proxy-tls\") pod \"84638809-bcd2-4539-b828-5496cddb9a17\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") "
Sep 17 17:58:16.064793 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.064687    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84638809-bcd2-4539-b828-5496cddb9a17-kserve-provision-location\") pod \"84638809-bcd2-4539-b828-5496cddb9a17\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") "
Sep 17 17:58:16.064793 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.064726    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-cabundle-cert\") pod \"84638809-bcd2-4539-b828-5496cddb9a17\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") "
Sep 17 17:58:16.064793 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.064753    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-kube-rbac-proxy-sar-config\") pod \"84638809-bcd2-4539-b828-5496cddb9a17\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") "
Sep 17 17:58:16.064793 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.064788    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccng7\" (UniqueName: \"kubernetes.io/projected/84638809-bcd2-4539-b828-5496cddb9a17-kube-api-access-ccng7\") pod \"84638809-bcd2-4539-b828-5496cddb9a17\" (UID: \"84638809-bcd2-4539-b828-5496cddb9a17\") "
Sep 17 17:58:16.065075 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.064987    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84638809-bcd2-4539-b828-5496cddb9a17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84638809-bcd2-4539-b828-5496cddb9a17" (UID: "84638809-bcd2-4539-b828-5496cddb9a17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:16.065141 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.065091    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84638809-bcd2-4539-b828-5496cddb9a17" (UID: "84638809-bcd2-4539-b828-5496cddb9a17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:16.065195 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.065155    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84638809-bcd2-4539-b828-5496cddb9a17" (UID: "84638809-bcd2-4539-b828-5496cddb9a17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:16.066924 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.066896    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84638809-bcd2-4539-b828-5496cddb9a17-kube-api-access-ccng7" (OuterVolumeSpecName: "kube-api-access-ccng7") pod "84638809-bcd2-4539-b828-5496cddb9a17" (UID: "84638809-bcd2-4539-b828-5496cddb9a17"). InnerVolumeSpecName "kube-api-access-ccng7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:16.067068 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.067053    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84638809-bcd2-4539-b828-5496cddb9a17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84638809-bcd2-4539-b828-5496cddb9a17" (UID: "84638809-bcd2-4539-b828-5496cddb9a17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:16.165779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.165753    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84638809-bcd2-4539-b828-5496cddb9a17-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:16.165779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.165776    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84638809-bcd2-4539-b828-5496cddb9a17-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:16.165779 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.165786    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:16.165990 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.165794    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84638809-bcd2-4539-b828-5496cddb9a17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:16.165990 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.165803    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccng7\" (UniqueName: \"kubernetes.io/projected/84638809-bcd2-4539-b828-5496cddb9a17-kube-api-access-ccng7\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:16.733686 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.733649    2585 generic.go:356] "Generic (PLEG): container finished" podID="84638809-bcd2-4539-b828-5496cddb9a17" containerID="069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922" exitCode=0
Sep 17 17:58:16.733867 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.733711    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"
Sep 17 17:58:16.733867 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.733741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerDied","Data":"069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922"}
Sep 17 17:58:16.733867 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.733833    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv" event={"ID":"84638809-bcd2-4539-b828-5496cddb9a17","Type":"ContainerDied","Data":"290881d0f55098a97798392ffb254cb34419a96c7467292efa37f004c5f171e0"}
Sep 17 17:58:16.733867 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.733836    2585 scope.go:117] "RemoveContainer" containerID="069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922"
Sep 17 17:58:16.742881 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.742861    2585 scope.go:117] "RemoveContainer" containerID="ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb"
Sep 17 17:58:16.750335 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.750318    2585 scope.go:117] "RemoveContainer" containerID="3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6"
Sep 17 17:58:16.759175 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.759148    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"]
Sep 17 17:58:16.760758 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.760740    2585 scope.go:117] "RemoveContainer" containerID="c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97"
Sep 17 17:58:16.761556 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.761533    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-wfstv"]
Sep 17 17:58:16.768028 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768012    2585 scope.go:117] "RemoveContainer" containerID="069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922"
Sep 17 17:58:16.768322 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:58:16.768295    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922\": container with ID starting with 069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922 not found: ID does not exist" containerID="069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922"
Sep 17 17:58:16.768396 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768325    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922"} err="failed to get container status \"069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922\": rpc error: code = NotFound desc = could not find container \"069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922\": container with ID starting with 069a2c291b0a9a851cb69702bf28e7c2742627a54b46dd1ba43815943c12d922 not found: ID does not exist"
Sep 17 17:58:16.768396 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768341    2585 scope.go:117] "RemoveContainer" containerID="ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb"
Sep 17 17:58:16.768576 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:58:16.768559    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb\": container with ID starting with ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb not found: ID does not exist" containerID="ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb"
Sep 17 17:58:16.768618 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768583    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb"} err="failed to get container status \"ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb\": rpc error: code = NotFound desc = could not find container \"ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb\": container with ID starting with ceb3bd736641fe66eba9564e8b52c11b313d543d2806ec9147e853889268ebdb not found: ID does not exist"
Sep 17 17:58:16.768618 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768598    2585 scope.go:117] "RemoveContainer" containerID="3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6"
Sep 17 17:58:16.768875 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:58:16.768856    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6\": container with ID starting with 3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6 not found: ID does not exist" containerID="3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6"
Sep 17 17:58:16.768938 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768879    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6"} err="failed to get container status \"3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6\": rpc error: code = NotFound desc = could not find container \"3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6\": container with ID starting with 3fb6e3ffb4ab61e03cbeb04ac9cb3450fded3f02098444eadf89981199f70bd6 not found: ID does not exist"
Sep 17 17:58:16.768938 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.768892    2585 scope.go:117] "RemoveContainer" containerID="c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97"
Sep 17 17:58:16.769135 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:58:16.769118    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97\": container with ID starting with c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97 not found: ID does not exist" containerID="c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97"
Sep 17 17:58:16.769172 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.769141    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97"} err="failed to get container status \"c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97\": rpc error: code = NotFound desc = could not find container \"c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97\": container with ID starting with c4e9a3b6f24180c9a347816b5c3bc3142990a66b6a3e9ff6fbb1c7b0b386bc97 not found: ID does not exist"
Sep 17 17:58:16.960431 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:16.960397    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84638809-bcd2-4539-b828-5496cddb9a17" path="/var/lib/kubelet/pods/84638809-bcd2-4539-b828-5496cddb9a17/volumes"
Sep 17 17:58:20.387982 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:20.387952    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:58:27.515641 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:27.515610    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"]
Sep 17 17:58:27.516015 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:27.515866    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" containerID="cri-o://a6840dcd91807f1b75c38f71ed0bebbcd4cb5fa77c6557b9c70e3b76d6dda356" gracePeriod=30
Sep 17 17:58:27.516015 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:27.515946    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kube-rbac-proxy" containerID="cri-o://5404dedac78c96ff01cbc373592406529c9f8e91525c8663fbdace2a9f629d4d" gracePeriod=30
Sep 17 17:58:27.786567 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:27.786487    2585 generic.go:356] "Generic (PLEG): container finished" podID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerID="5404dedac78c96ff01cbc373592406529c9f8e91525c8663fbdace2a9f629d4d" exitCode=2
Sep 17 17:58:27.786567 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:27.786546    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerDied","Data":"5404dedac78c96ff01cbc373592406529c9f8e91525c8663fbdace2a9f629d4d"}
Sep 17 17:58:30.383628 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:30.383582    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 17:58:30.388272 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:30.388246    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:58:32.713943 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.713902    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"]
Sep 17 17:58:32.714451 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714432    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kube-rbac-proxy"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714454    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kube-rbac-proxy"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714470    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kserve-container"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714478    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kserve-container"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714493    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714502    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714517    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy"
Sep 17 17:58:32.714533 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714525    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714549    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="storage-initializer"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714557    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="storage-initializer"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714568    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714576    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714663    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="agent"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714676    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kube-rbac-proxy"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714686    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="84638809-bcd2-4539-b828-5496cddb9a17" containerName="kserve-container"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714694    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kserve-container"
Sep 17 17:58:32.714934 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.714707    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf57785-d01b-46a1-8dcd-6e0195d0d4a5" containerName="kube-rbac-proxy"
Sep 17 17:58:32.718231 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.718211    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.720656 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.720630    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 17:58:32.720656 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.720632    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:58:32.727696 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.727677    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"]
Sep 17 17:58:32.797688 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.797656    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.797854 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.797706    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbzf\" (UniqueName: \"kubernetes.io/projected/a59f9962-c156-4615-8057-d8130b630bc0-kube-api-access-8zbzf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.797854 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.797737    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.797854 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.797836    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.797992 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.797884    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a59f9962-c156-4615-8057-d8130b630bc0-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.812412 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.812383    2585 generic.go:356] "Generic (PLEG): container finished" podID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerID="a6840dcd91807f1b75c38f71ed0bebbcd4cb5fa77c6557b9c70e3b76d6dda356" exitCode=0
Sep 17 17:58:32.812522 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.812450    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerDied","Data":"a6840dcd91807f1b75c38f71ed0bebbcd4cb5fa77c6557b9c70e3b76d6dda356"}
Sep 17 17:58:32.898475 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.898453    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.898572 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.898486    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.898572 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.898508    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a59f9962-c156-4615-8057-d8130b630bc0-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.898679 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.898574    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.898679 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:58:32.898610    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:58:32.898679 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.898616    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbzf\" (UniqueName: \"kubernetes.io/projected/a59f9962-c156-4615-8057-d8130b630bc0-kube-api-access-8zbzf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.898855 ip-10-0-138-144 kubenswrapper[2585]: E0917 17:58:32.898684    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls podName:a59f9962-c156-4615-8057-d8130b630bc0 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:33.398653029 +0000 UTC m=+1046.589990709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" (UID: "a59f9962-c156-4615-8057-d8130b630bc0") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:58:32.898994 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.898973    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a59f9962-c156-4615-8057-d8130b630bc0-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.899218 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.899199    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.899253 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.899220    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:32.906869 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:32.906845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbzf\" (UniqueName: \"kubernetes.io/projected/a59f9962-c156-4615-8057-d8130b630bc0-kube-api-access-8zbzf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:33.159932 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.159908    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:58:33.200499 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200470    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-cabundle-cert\") pod \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") "
Sep 17 17:58:33.200651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200518    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a81288-f843-4d87-ba2b-cdd5d14660fb-proxy-tls\") pod \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") "
Sep 17 17:58:33.200651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200548    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-rbac-proxy-sar-config\") pod \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") "
Sep 17 17:58:33.200651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200603    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kserve-provision-location\") pod \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") "
Sep 17 17:58:33.200651 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200646    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pgdz\" (UniqueName: \"kubernetes.io/projected/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-api-access-8pgdz\") pod \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\" (UID: \"b5a81288-f843-4d87-ba2b-cdd5d14660fb\") "
Sep 17 17:58:33.200948 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200900    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5a81288-f843-4d87-ba2b-cdd5d14660fb" (UID: "b5a81288-f843-4d87-ba2b-cdd5d14660fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:33.201051 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.200999    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5a81288-f843-4d87-ba2b-cdd5d14660fb" (UID: "b5a81288-f843-4d87-ba2b-cdd5d14660fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:33.201051 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.201022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5a81288-f843-4d87-ba2b-cdd5d14660fb" (UID: "b5a81288-f843-4d87-ba2b-cdd5d14660fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:33.202697 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.202678    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a81288-f843-4d87-ba2b-cdd5d14660fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5a81288-f843-4d87-ba2b-cdd5d14660fb" (UID: "b5a81288-f843-4d87-ba2b-cdd5d14660fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:33.202765 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.202751    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-api-access-8pgdz" (OuterVolumeSpecName: "kube-api-access-8pgdz") pod "b5a81288-f843-4d87-ba2b-cdd5d14660fb" (UID: "b5a81288-f843-4d87-ba2b-cdd5d14660fb"). InnerVolumeSpecName "kube-api-access-8pgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:33.301335 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.301306    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgdz\" (UniqueName: \"kubernetes.io/projected/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-api-access-8pgdz\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:33.301335 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.301331    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:33.301498 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.301344    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a81288-f843-4d87-ba2b-cdd5d14660fb-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:33.301498 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.301355    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:33.301498 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.301367    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a81288-f843-4d87-ba2b-cdd5d14660fb-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 17:58:33.402031 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.402005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:33.404426 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.404409    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:33.630899 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.630870    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:33.818552 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.818524    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"
Sep 17 17:58:33.819023 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.818520    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq" event={"ID":"b5a81288-f843-4d87-ba2b-cdd5d14660fb","Type":"ContainerDied","Data":"f9ada4e8dfd2c50a09e180f6323e1496f0a03ee7949e90dd948211223e944dd1"}
Sep 17 17:58:33.819023 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.818661    2585 scope.go:117] "RemoveContainer" containerID="5404dedac78c96ff01cbc373592406529c9f8e91525c8663fbdace2a9f629d4d"
Sep 17 17:58:33.827358 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.827336    2585 scope.go:117] "RemoveContainer" containerID="a6840dcd91807f1b75c38f71ed0bebbcd4cb5fa77c6557b9c70e3b76d6dda356"
Sep 17 17:58:33.834739 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.834723    2585 scope.go:117] "RemoveContainer" containerID="86523bf2f707495756ddbe1d3f1c04d9049194851e52e5a2e2d24f774b042709"
Sep 17 17:58:33.845966 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.845943    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"]
Sep 17 17:58:33.850152 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.850128    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6zghq"]
Sep 17 17:58:33.926104 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:33.926085    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"]
Sep 17 17:58:33.928736 ip-10-0-138-144 kubenswrapper[2585]: W0917 17:58:33.928679    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59f9962_c156_4615_8057_d8130b630bc0.slice/crio-1f57d11a86414725308d65cab8585ff566f092c51b749c34f259bf498bae781b WatchSource:0}: Error finding container 1f57d11a86414725308d65cab8585ff566f092c51b749c34f259bf498bae781b: Status 404 returned error can't find the container with id 1f57d11a86414725308d65cab8585ff566f092c51b749c34f259bf498bae781b
Sep 17 17:58:34.825354 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:34.825315    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerStarted","Data":"2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d"}
Sep 17 17:58:34.825868 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:34.825387    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerStarted","Data":"1f57d11a86414725308d65cab8585ff566f092c51b749c34f259bf498bae781b"}
Sep 17 17:58:34.959615 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:34.959582    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" path="/var/lib/kubelet/pods/b5a81288-f843-4d87-ba2b-cdd5d14660fb/volumes"
Sep 17 17:58:38.847092 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:38.847053    2585 generic.go:356] "Generic (PLEG): container finished" podID="a59f9962-c156-4615-8057-d8130b630bc0" containerID="2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d" exitCode=0
Sep 17 17:58:38.847465 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:38.847128    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerDied","Data":"2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d"}
Sep 17 17:58:39.853626 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:39.853595    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerStarted","Data":"35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c"}
Sep 17 17:58:39.854010 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:39.853659    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerStarted","Data":"287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7"}
Sep 17 17:58:39.854010 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:39.853868    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:39.854010 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:39.853911    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:58:39.874987 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:39.874921    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podStartSLOduration=4.006586782 podStartE2EDuration="7.874907035s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868320253s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:39.87377123 +0000 UTC m=+1053.065108908" watchObservedRunningTime="2026-09-17 17:58:39.874907035 +0000 UTC m=+1053.066244719"
Sep 17 17:58:45.865242 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:58:45.865215    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 17:59:15.866906 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:59:15.866865    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:59:25.866573 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:59:25.866532    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:59:35.865735 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:59:35.865694    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:59:45.866387 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:59:45.866339    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:59:48.956774 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:59:48.956734    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:59:59.398324 ip-10-0-138-144 kubenswrapper[2585]: I0917 17:59:59.398294    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 18:00:02.870396 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:02.870358    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"]
Sep 17 18:00:02.870868 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:02.870670    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container" containerID="cri-o://287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7" gracePeriod=30
Sep 17 18:00:02.870868 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:02.870742    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kube-rbac-proxy" containerID="cri-o://35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c" gracePeriod=30
Sep 17 18:00:03.239100 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:03.239024    2585 generic.go:356] "Generic (PLEG): container finished" podID="a59f9962-c156-4615-8057-d8130b630bc0" containerID="35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c" exitCode=2
Sep 17 18:00:03.239100 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:03.239092    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerDied","Data":"35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c"}
Sep 17 18:00:05.860069 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:05.860029    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 18:00:07.611882 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.611847    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 18:00:07.755213 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755140    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zbzf\" (UniqueName: \"kubernetes.io/projected/a59f9962-c156-4615-8057-d8130b630bc0-kube-api-access-8zbzf\") pod \"a59f9962-c156-4615-8057-d8130b630bc0\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") "
Sep 17 18:00:07.755213 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755175    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a59f9962-c156-4615-8057-d8130b630bc0-kserve-provision-location\") pod \"a59f9962-c156-4615-8057-d8130b630bc0\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") "
Sep 17 18:00:07.755418 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755223    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-kube-rbac-proxy-sar-config\") pod \"a59f9962-c156-4615-8057-d8130b630bc0\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") "
Sep 17 18:00:07.755418 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755328    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-cabundle-cert\") pod \"a59f9962-c156-4615-8057-d8130b630bc0\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") "
Sep 17 18:00:07.755418 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755378    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls\") pod \"a59f9962-c156-4615-8057-d8130b630bc0\" (UID: \"a59f9962-c156-4615-8057-d8130b630bc0\") "
Sep 17 18:00:07.755589 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755541    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59f9962-c156-4615-8057-d8130b630bc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a59f9962-c156-4615-8057-d8130b630bc0" (UID: "a59f9962-c156-4615-8057-d8130b630bc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:07.755589 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755573    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a59f9962-c156-4615-8057-d8130b630bc0" (UID: "a59f9962-c156-4615-8057-d8130b630bc0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:07.755675 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755639    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:00:07.755675 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755654    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a59f9962-c156-4615-8057-d8130b630bc0-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:00:07.755764 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.755683    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a59f9962-c156-4615-8057-d8130b630bc0" (UID: "a59f9962-c156-4615-8057-d8130b630bc0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:07.757357 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.757338    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59f9962-c156-4615-8057-d8130b630bc0-kube-api-access-8zbzf" (OuterVolumeSpecName: "kube-api-access-8zbzf") pod "a59f9962-c156-4615-8057-d8130b630bc0" (UID: "a59f9962-c156-4615-8057-d8130b630bc0"). InnerVolumeSpecName "kube-api-access-8zbzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:07.757513 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.757497    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a59f9962-c156-4615-8057-d8130b630bc0" (UID: "a59f9962-c156-4615-8057-d8130b630bc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:07.856784 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.856759    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a59f9962-c156-4615-8057-d8130b630bc0-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:00:07.856903 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.856786    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbzf\" (UniqueName: \"kubernetes.io/projected/a59f9962-c156-4615-8057-d8130b630bc0-kube-api-access-8zbzf\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:00:07.856903 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:07.856800    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a59f9962-c156-4615-8057-d8130b630bc0-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:00:08.068391 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068311    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"]
Sep 17 18:00:08.068662 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068650    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kube-rbac-proxy"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068664    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kube-rbac-proxy"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068673    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kube-rbac-proxy"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068679    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kube-rbac-proxy"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068687    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068693    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068705    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="storage-initializer"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068711    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="storage-initializer"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068717    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="storage-initializer"
Sep 17 18:00:08.068726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068721    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="storage-initializer"
Sep 17 18:00:08.069020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068731    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container"
Sep 17 18:00:08.069020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068736    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container"
Sep 17 18:00:08.069020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068804    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kserve-container"
Sep 17 18:00:08.069020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068826    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kserve-container"
Sep 17 18:00:08.069020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068832    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59f9962-c156-4615-8057-d8130b630bc0" containerName="kube-rbac-proxy"
Sep 17 18:00:08.069020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.068841    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a81288-f843-4d87-ba2b-cdd5d14660fb" containerName="kube-rbac-proxy"
Sep 17 18:00:08.075615 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.075595    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.078128 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.078104    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 18:00:08.078236 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.078129    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:00:08.093097 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.093072    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"]
Sep 17 18:00:08.159137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.159102    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.159137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.159139    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.159298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.159170    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a94740-2043-4121-98d3-5441b7089bf5-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.159298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.159248    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.159383 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.159350    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd95d\" (UniqueName: \"kubernetes.io/projected/42a94740-2043-4121-98d3-5441b7089bf5-kube-api-access-gd95d\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.260354 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.260324    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd95d\" (UniqueName: \"kubernetes.io/projected/42a94740-2043-4121-98d3-5441b7089bf5-kube-api-access-gd95d\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.260501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.260362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.260501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.260384    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.260627 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:00:08.260500    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 18:00:08.260627 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.260516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a94740-2043-4121-98d3-5441b7089bf5-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.260627 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:00:08.260572    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls podName:42a94740-2043-4121-98d3-5441b7089bf5 nodeName:}" failed. No retries permitted until 2026-09-17 18:00:08.760550433 +0000 UTC m=+1141.951888098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" (UID: "42a94740-2043-4121-98d3-5441b7089bf5") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 18:00:08.260780 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.260625    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.260916 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.260886    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a94740-2043-4121-98d3-5441b7089bf5-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.261046 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.261029    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.261253 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.261233    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.264029 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.264006    2585 generic.go:356] "Generic (PLEG): container finished" podID="a59f9962-c156-4615-8057-d8130b630bc0" containerID="287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7" exitCode=0
Sep 17 18:00:08.264125 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.264072    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"
Sep 17 18:00:08.264125 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.264089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerDied","Data":"287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7"}
Sep 17 18:00:08.264213 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.264161    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb" event={"ID":"a59f9962-c156-4615-8057-d8130b630bc0","Type":"ContainerDied","Data":"1f57d11a86414725308d65cab8585ff566f092c51b749c34f259bf498bae781b"}
Sep 17 18:00:08.264213 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.264198    2585 scope.go:117] "RemoveContainer" containerID="35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c"
Sep 17 18:00:08.269767 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.269743    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd95d\" (UniqueName: \"kubernetes.io/projected/42a94740-2043-4121-98d3-5441b7089bf5-kube-api-access-gd95d\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.275982 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.275954    2585 scope.go:117] "RemoveContainer" containerID="287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7"
Sep 17 18:00:08.283940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.283924    2585 scope.go:117] "RemoveContainer" containerID="2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d"
Sep 17 18:00:08.289177 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.289158    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"]
Sep 17 18:00:08.291785 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.291763    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-s5bfb"]
Sep 17 18:00:08.292891 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.292873    2585 scope.go:117] "RemoveContainer" containerID="35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c"
Sep 17 18:00:08.293161 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:00:08.293143    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c\": container with ID starting with 35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c not found: ID does not exist" containerID="35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c"
Sep 17 18:00:08.293234 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.293173    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c"} err="failed to get container status \"35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c\": rpc error: code = NotFound desc = could not find container \"35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c\": container with ID starting with 35e550a42b13e12bacb37fe3e05a19c28a1bc63672f289fe247f5743c391dd5c not found: ID does not exist"
Sep 17 18:00:08.293234 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.293197    2585 scope.go:117] "RemoveContainer" containerID="287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7"
Sep 17 18:00:08.293485 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:00:08.293467    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7\": container with ID starting with 287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7 not found: ID does not exist" containerID="287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7"
Sep 17 18:00:08.293521 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.293491    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7"} err="failed to get container status \"287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7\": rpc error: code = NotFound desc = could not find container \"287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7\": container with ID starting with 287a372fe4a14b06c00b79a119e2993106906c399e12411402d0218d151391a7 not found: ID does not exist"
Sep 17 18:00:08.293521 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.293507    2585 scope.go:117] "RemoveContainer" containerID="2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d"
Sep 17 18:00:08.293754 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:00:08.293736    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d\": container with ID starting with 2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d not found: ID does not exist" containerID="2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d"
Sep 17 18:00:08.293798 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.293769    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d"} err="failed to get container status \"2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d\": rpc error: code = NotFound desc = could not find container \"2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d\": container with ID starting with 2c96c09d78d05cf73290e2c3acc0dd6b41b4ee56e4d970e52fd4e00c56c65a1d not found: ID does not exist"
Sep 17 18:00:08.765559 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.765518    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.768004 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.767985    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:08.959872 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.959845    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a59f9962-c156-4615-8057-d8130b630bc0" path="/var/lib/kubelet/pods/a59f9962-c156-4615-8057-d8130b630bc0/volumes"
Sep 17 18:00:08.989438 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:08.989412    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:09.265312 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:09.265288    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"]
Sep 17 18:00:09.267735 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:00:09.267712    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a94740_2043_4121_98d3_5441b7089bf5.slice/crio-25e95b44d4c8820b00472b659fa0e6ea63bb61044ae02ba726f11c60ef0f5709 WatchSource:0}: Error finding container 25e95b44d4c8820b00472b659fa0e6ea63bb61044ae02ba726f11c60ef0f5709: Status 404 returned error can't find the container with id 25e95b44d4c8820b00472b659fa0e6ea63bb61044ae02ba726f11c60ef0f5709
Sep 17 18:00:10.276864 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:10.276827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerStarted","Data":"b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f"}
Sep 17 18:00:10.277229 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:10.276887    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerStarted","Data":"25e95b44d4c8820b00472b659fa0e6ea63bb61044ae02ba726f11c60ef0f5709"}
Sep 17 18:00:14.296441 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:14.296403    2585 generic.go:356] "Generic (PLEG): container finished" podID="42a94740-2043-4121-98d3-5441b7089bf5" containerID="b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f" exitCode=0
Sep 17 18:00:14.296919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:14.296475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerDied","Data":"b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f"}
Sep 17 18:00:15.303366 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:15.303332    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerStarted","Data":"f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95"}
Sep 17 18:00:15.303772 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:15.303395    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerStarted","Data":"7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff"}
Sep 17 18:00:15.303772 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:15.303618    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:15.303772 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:15.303673    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:15.333291 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:15.333246    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podStartSLOduration=3.365164441 podStartE2EDuration="7.333233977s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.968069536s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:15.330884124 +0000 UTC m=+1148.522221811" watchObservedRunningTime="2026-09-17 18:00:15.333233977 +0000 UTC m=+1148.524571719"
Sep 17 18:00:21.315697 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:21.315668    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:00:27.330872 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.330836    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"]
Sep 17 18:00:27.335197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.335177    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.337825 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.337791    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:00:27.337825 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.337805    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 18:00:27.344520 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.344491    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"]
Sep 17 18:00:27.519613 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.519577    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-api-access-mb2xl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.519775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.519619    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/386eb205-daf8-45b1-866a-b79e5bc50c72-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.519775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.519645    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.519907 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.519766    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386eb205-daf8-45b1-866a-b79e5bc50c72-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.519907 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.519834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.620903 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.620874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386eb205-daf8-45b1-866a-b79e5bc50c72-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621068 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.620924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621068 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.620963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-api-access-mb2xl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621068 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.620999    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/386eb205-daf8-45b1-866a-b79e5bc50c72-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621068 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.621022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621479 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.621452    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/386eb205-daf8-45b1-866a-b79e5bc50c72-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621635 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.621616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.621703 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.621662    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.623545 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.623527    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386eb205-daf8-45b1-866a-b79e5bc50c72-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.629305 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.629282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-api-access-mb2xl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-gwk46\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.647120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.647089    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:00:27.919996 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:27.919968    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"]
Sep 17 18:00:27.921402 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:00:27.921375    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386eb205_daf8_45b1_866a_b79e5bc50c72.slice/crio-9b2238f3843647216fbbd6d8d541eaf13b4fe0db25d0e7b15f343f996343da7d WatchSource:0}: Error finding container 9b2238f3843647216fbbd6d8d541eaf13b4fe0db25d0e7b15f343f996343da7d: Status 404 returned error can't find the container with id 9b2238f3843647216fbbd6d8d541eaf13b4fe0db25d0e7b15f343f996343da7d
Sep 17 18:00:28.368744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:28.368709    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerStarted","Data":"f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355"}
Sep 17 18:00:28.369116 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:28.368769    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerStarted","Data":"9b2238f3843647216fbbd6d8d541eaf13b4fe0db25d0e7b15f343f996343da7d"}
Sep 17 18:00:51.316800 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:51.316760    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:00:57.505911 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:57.505871    2585 generic.go:356] "Generic (PLEG): container finished" podID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerID="f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355" exitCode=0
Sep 17 18:00:57.506280 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:57.505939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerDied","Data":"f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355"}
Sep 17 18:00:57.507302 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:00:57.507287    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:01:01.316936 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:01.316896    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:08.983854 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:08.983825    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:01:08.984786 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:08.984766    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:01:11.316239 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:11.316196    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:21.316904 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:21.316858    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:31.316446 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:31.316409    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:41.990185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:41.990146    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:01:48.129860 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:48.129826    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"]
Sep 17 18:01:48.130372 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:48.130178    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" containerID="cri-o://7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff" gracePeriod=30
Sep 17 18:01:48.130372 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:48.130207    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kube-rbac-proxy" containerID="cri-o://f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95" gracePeriod=30
Sep 17 18:01:48.774729 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:48.774696    2585 generic.go:356] "Generic (PLEG): container finished" podID="42a94740-2043-4121-98d3-5441b7089bf5" containerID="f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95" exitCode=2
Sep 17 18:01:48.774912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:48.774769    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerDied","Data":"f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95"}
Sep 17 18:01:51.311535 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:51.311471    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 18:01:51.316932 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:51.316899    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:53.405025 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.404996    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:01:53.438853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.438827    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd95d\" (UniqueName: \"kubernetes.io/projected/42a94740-2043-4121-98d3-5441b7089bf5-kube-api-access-gd95d\") pod \"42a94740-2043-4121-98d3-5441b7089bf5\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") "
Sep 17 18:01:53.438994 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.438867    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls\") pod \"42a94740-2043-4121-98d3-5441b7089bf5\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") "
Sep 17 18:01:53.438994 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.438932    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-cabundle-cert\") pod \"42a94740-2043-4121-98d3-5441b7089bf5\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") "
Sep 17 18:01:53.438994 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.438960    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a94740-2043-4121-98d3-5441b7089bf5-kserve-provision-location\") pod \"42a94740-2043-4121-98d3-5441b7089bf5\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") "
Sep 17 18:01:53.439105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.439009    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-kube-rbac-proxy-sar-config\") pod \"42a94740-2043-4121-98d3-5441b7089bf5\" (UID: \"42a94740-2043-4121-98d3-5441b7089bf5\") "
Sep 17 18:01:53.439671 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.439294    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42a94740-2043-4121-98d3-5441b7089bf5" (UID: "42a94740-2043-4121-98d3-5441b7089bf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:53.439671 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.439294    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a94740-2043-4121-98d3-5441b7089bf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42a94740-2043-4121-98d3-5441b7089bf5" (UID: "42a94740-2043-4121-98d3-5441b7089bf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:53.439671 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.439540    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42a94740-2043-4121-98d3-5441b7089bf5" (UID: "42a94740-2043-4121-98d3-5441b7089bf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:53.441197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.441176    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42a94740-2043-4121-98d3-5441b7089bf5" (UID: "42a94740-2043-4121-98d3-5441b7089bf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:53.441316 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.441293    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a94740-2043-4121-98d3-5441b7089bf5-kube-api-access-gd95d" (OuterVolumeSpecName: "kube-api-access-gd95d") pod "42a94740-2043-4121-98d3-5441b7089bf5" (UID: "42a94740-2043-4121-98d3-5441b7089bf5"). InnerVolumeSpecName "kube-api-access-gd95d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:53.540417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.540384    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:01:53.540417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.540410    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a94740-2043-4121-98d3-5441b7089bf5-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:01:53.540417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.540422    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a94740-2043-4121-98d3-5441b7089bf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:01:53.540667 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.540430    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd95d\" (UniqueName: \"kubernetes.io/projected/42a94740-2043-4121-98d3-5441b7089bf5-kube-api-access-gd95d\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:01:53.540667 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.540439    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a94740-2043-4121-98d3-5441b7089bf5-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:01:53.802162 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.802083    2585 generic.go:356] "Generic (PLEG): container finished" podID="42a94740-2043-4121-98d3-5441b7089bf5" containerID="7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff" exitCode=0
Sep 17 18:01:53.802162 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.802136    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"
Sep 17 18:01:53.802401 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.802133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerDied","Data":"7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff"}
Sep 17 18:01:53.802401 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.802293    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv" event={"ID":"42a94740-2043-4121-98d3-5441b7089bf5","Type":"ContainerDied","Data":"25e95b44d4c8820b00472b659fa0e6ea63bb61044ae02ba726f11c60ef0f5709"}
Sep 17 18:01:53.802401 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.802335    2585 scope.go:117] "RemoveContainer" containerID="f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95"
Sep 17 18:01:53.814960 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.814942    2585 scope.go:117] "RemoveContainer" containerID="7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff"
Sep 17 18:01:53.823289 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.823272    2585 scope.go:117] "RemoveContainer" containerID="b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f"
Sep 17 18:01:53.828533 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.828508    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"]
Sep 17 18:01:53.833125 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.833032    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-85dcv"]
Sep 17 18:01:53.833194 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.833156    2585 scope.go:117] "RemoveContainer" containerID="f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95"
Sep 17 18:01:53.833517 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:01:53.833498    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95\": container with ID starting with f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95 not found: ID does not exist" containerID="f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95"
Sep 17 18:01:53.833622 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.833528    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95"} err="failed to get container status \"f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95\": rpc error: code = NotFound desc = could not find container \"f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95\": container with ID starting with f2c37aed6debb016c8f90f02b76bd35d901aadca253c173791089029bd9f9d95 not found: ID does not exist"
Sep 17 18:01:53.833622 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.833550    2585 scope.go:117] "RemoveContainer" containerID="7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff"
Sep 17 18:01:53.834232 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:01:53.834204    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff\": container with ID starting with 7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff not found: ID does not exist" containerID="7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff"
Sep 17 18:01:53.834319 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.834242    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff"} err="failed to get container status \"7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff\": rpc error: code = NotFound desc = could not find container \"7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff\": container with ID starting with 7f2c008412169b7eb88a9bf515f4e9d93ff5bb800790c7d5a377e8dd6c7349ff not found: ID does not exist"
Sep 17 18:01:53.834319 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.834265    2585 scope.go:117] "RemoveContainer" containerID="b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f"
Sep 17 18:01:53.834562 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:01:53.834536    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f\": container with ID starting with b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f not found: ID does not exist" containerID="b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f"
Sep 17 18:01:53.834618 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:53.834570    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f"} err="failed to get container status \"b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f\": rpc error: code = NotFound desc = could not find container \"b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f\": container with ID starting with b3143101086d0f63718df1c6c1be66073951dc4fe4971a31bf78a2efedc5362f not found: ID does not exist"
Sep 17 18:01:54.959419 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:01:54.959384    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a94740-2043-4121-98d3-5441b7089bf5" path="/var/lib/kubelet/pods/42a94740-2043-4121-98d3-5441b7089bf5/volumes"
Sep 17 18:03:23.272601 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:03:23.272561    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerStarted","Data":"221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c"}
Sep 17 18:03:23.273165 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:03:23.272627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerStarted","Data":"6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744"}
Sep 17 18:03:23.273165 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:03:23.272655    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:03:23.273165 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:03:23.272787    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:03:23.298417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:03:23.298339    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" podStartSLOduration=2.628249779 podStartE2EDuration="2m56.298317445s" totalImagesPullingTime="2m24.70994236s" totalInitContainerRuntime="28.960125306s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:23.295704545 +0000 UTC m=+1336.487042260" watchObservedRunningTime="2026-09-17 18:03:23.298317445 +0000 UTC m=+1336.489655132"
Sep 17 18:03:29.283148 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:03:29.283121    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:04:02.781402 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.781359    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"]
Sep 17 18:04:02.782047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.781957    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="storage-initializer"
Sep 17 18:04:02.782047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.781978    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="storage-initializer"
Sep 17 18:04:02.782047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.781992    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container"
Sep 17 18:04:02.782047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.782000    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container"
Sep 17 18:04:02.782047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.782036    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kube-rbac-proxy"
Sep 17 18:04:02.782047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.782044    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kube-rbac-proxy"
Sep 17 18:04:02.782379 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.782136    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kserve-container"
Sep 17 18:04:02.782379 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.782152    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a94740-2043-4121-98d3-5441b7089bf5" containerName="kube-rbac-proxy"
Sep 17 18:04:02.784509 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.784490    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.787270 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.787244    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 18:04:02.787270 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.787261    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 18:04:02.795537 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.795518    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"]
Sep 17 18:04:02.866028 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.865996    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zm9\" (UniqueName: \"kubernetes.io/projected/c7966fb4-0e58-406c-9c60-979326232271-kube-api-access-g9zm9\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.866167 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.866038    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.866167 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.866112    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-cabundle-cert\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.866290 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.866186    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7966fb4-0e58-406c-9c60-979326232271-kserve-provision-location\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.866290 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.866232    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.966619 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.966585    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7966fb4-0e58-406c-9c60-979326232271-kserve-provision-location\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.966619 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.966622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.966879 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.966645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zm9\" (UniqueName: \"kubernetes.io/projected/c7966fb4-0e58-406c-9c60-979326232271-kube-api-access-g9zm9\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.966879 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.966675    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.966879 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.966720    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-cabundle-cert\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.967002 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:04:02.966892    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 18:04:02.967002 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:04:02.966979    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls podName:c7966fb4-0e58-406c-9c60-979326232271 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:03.466958336 +0000 UTC m=+1376.658296006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls") pod "isvc-sklearn-predictor-56c47844df-cdfjt" (UID: "c7966fb4-0e58-406c-9c60-979326232271") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 18:04:02.967090 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.967078    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7966fb4-0e58-406c-9c60-979326232271-kserve-provision-location\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.967471 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.967451    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.967512 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.967486    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-cabundle-cert\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:02.976106 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:02.976087    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zm9\" (UniqueName: \"kubernetes.io/projected/c7966fb4-0e58-406c-9c60-979326232271-kube-api-access-g9zm9\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:03.470143 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:03.470113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:03.472763 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:03.472741    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-cdfjt\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:03.696214 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:03.696170    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:03.970011 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:03.969988    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"]
Sep 17 18:04:03.971912 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:04:03.971888    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7966fb4_0e58_406c_9c60_979326232271.slice/crio-29b340d4f2d55b060bbb602823c6689a4a9624d9563c0ca14a645fc224b6a1ee WatchSource:0}: Error finding container 29b340d4f2d55b060bbb602823c6689a4a9624d9563c0ca14a645fc224b6a1ee: Status 404 returned error can't find the container with id 29b340d4f2d55b060bbb602823c6689a4a9624d9563c0ca14a645fc224b6a1ee
Sep 17 18:04:04.469278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:04.469237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerStarted","Data":"a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d"}
Sep 17 18:04:04.469469 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:04.469309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerStarted","Data":"29b340d4f2d55b060bbb602823c6689a4a9624d9563c0ca14a645fc224b6a1ee"}
Sep 17 18:04:08.489387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:08.489347    2585 generic.go:356] "Generic (PLEG): container finished" podID="c7966fb4-0e58-406c-9c60-979326232271" containerID="a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d" exitCode=0
Sep 17 18:04:08.489876 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:08.489428    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerDied","Data":"a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d"}
Sep 17 18:04:09.496497 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:09.496462    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerStarted","Data":"14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709"}
Sep 17 18:04:09.496894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:09.496529    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerStarted","Data":"bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131"}
Sep 17 18:04:09.496894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:09.496753    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:09.520522 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:09.520468    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podStartSLOduration=3.716278313 podStartE2EDuration="7.520454014s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.804175701s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:09.518403487 +0000 UTC m=+1382.709741173" watchObservedRunningTime="2026-09-17 18:04:09.520454014 +0000 UTC m=+1382.711791713"
Sep 17 18:04:10.501864 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:10.501831    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:10.503138 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:10.503108    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:04:11.506468 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:11.506421    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:04:16.510946 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:16.510916    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:04:16.511521 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:16.511494    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:04:26.511827 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:26.511782    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:04:36.512499 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:36.512462    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:04:46.512447 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:46.512411    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:04:56.511564 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:04:56.511474    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:05:01.987545 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:01.987513    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:05:06.512328 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:06.512292    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:05:12.995991 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:12.995941    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:05:13.720005 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:13.719974    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"]
Sep 17 18:05:13.720239 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:13.720212    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kserve-container" containerID="cri-o://6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744" gracePeriod=30
Sep 17 18:05:13.720338 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:13.720293    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kube-rbac-proxy" containerID="cri-o://221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c" gracePeriod=30
Sep 17 18:05:14.278243 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:14.278202    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 18:05:14.816744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:14.816713    2585 generic.go:356] "Generic (PLEG): container finished" podID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerID="221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c" exitCode=2
Sep 17 18:05:14.816938 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:14.816777    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerDied","Data":"221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c"}
Sep 17 18:05:15.068853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.068790    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:05:15.148701 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.148669    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-rbac-proxy-sar-config\") pod \"386eb205-daf8-45b1-866a-b79e5bc50c72\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") "
Sep 17 18:05:15.148894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.148716    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/386eb205-daf8-45b1-866a-b79e5bc50c72-kserve-provision-location\") pod \"386eb205-daf8-45b1-866a-b79e5bc50c72\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") "
Sep 17 18:05:15.148894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.148743    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-cabundle-cert\") pod \"386eb205-daf8-45b1-866a-b79e5bc50c72\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") "
Sep 17 18:05:15.148894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.148783    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386eb205-daf8-45b1-866a-b79e5bc50c72-proxy-tls\") pod \"386eb205-daf8-45b1-866a-b79e5bc50c72\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") "
Sep 17 18:05:15.148894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.148854    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-api-access-mb2xl\") pod \"386eb205-daf8-45b1-866a-b79e5bc50c72\" (UID: \"386eb205-daf8-45b1-866a-b79e5bc50c72\") "
Sep 17 18:05:15.149143 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.149019    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "386eb205-daf8-45b1-866a-b79e5bc50c72" (UID: "386eb205-daf8-45b1-866a-b79e5bc50c72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:15.149143 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.149129    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "386eb205-daf8-45b1-866a-b79e5bc50c72" (UID: "386eb205-daf8-45b1-866a-b79e5bc50c72"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:15.151052 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.151026    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-api-access-mb2xl" (OuterVolumeSpecName: "kube-api-access-mb2xl") pod "386eb205-daf8-45b1-866a-b79e5bc50c72" (UID: "386eb205-daf8-45b1-866a-b79e5bc50c72"). InnerVolumeSpecName "kube-api-access-mb2xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:15.151172 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.151124    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386eb205-daf8-45b1-866a-b79e5bc50c72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "386eb205-daf8-45b1-866a-b79e5bc50c72" (UID: "386eb205-daf8-45b1-866a-b79e5bc50c72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:15.152210 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.152188    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386eb205-daf8-45b1-866a-b79e5bc50c72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "386eb205-daf8-45b1-866a-b79e5bc50c72" (UID: "386eb205-daf8-45b1-866a-b79e5bc50c72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:15.249559 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.249533    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-api-access-mb2xl\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:15.249559 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.249555    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:15.249689 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.249567    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/386eb205-daf8-45b1-866a-b79e5bc50c72-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:15.249689 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.249576    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/386eb205-daf8-45b1-866a-b79e5bc50c72-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:15.249689 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.249584    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386eb205-daf8-45b1-866a-b79e5bc50c72-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:15.824075 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.824036    2585 generic.go:356] "Generic (PLEG): container finished" podID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerID="6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744" exitCode=0
Sep 17 18:05:15.824479 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.824098    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"
Sep 17 18:05:15.824479 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.824117    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerDied","Data":"6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744"}
Sep 17 18:05:15.824479 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.824181    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46" event={"ID":"386eb205-daf8-45b1-866a-b79e5bc50c72","Type":"ContainerDied","Data":"9b2238f3843647216fbbd6d8d541eaf13b4fe0db25d0e7b15f343f996343da7d"}
Sep 17 18:05:15.824479 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.824208    2585 scope.go:117] "RemoveContainer" containerID="221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c"
Sep 17 18:05:15.835365 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.835344    2585 scope.go:117] "RemoveContainer" containerID="6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744"
Sep 17 18:05:15.843331 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.843310    2585 scope.go:117] "RemoveContainer" containerID="f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355"
Sep 17 18:05:15.850683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.850659    2585 scope.go:117] "RemoveContainer" containerID="221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c"
Sep 17 18:05:15.850982 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:15.850961    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c\": container with ID starting with 221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c not found: ID does not exist" containerID="221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c"
Sep 17 18:05:15.851072 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.850995    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c"} err="failed to get container status \"221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c\": rpc error: code = NotFound desc = could not find container \"221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c\": container with ID starting with 221e41793ef085cb428a176656f43c7d0b9e77b8e0838f43a0359efc0bebcc1c not found: ID does not exist"
Sep 17 18:05:15.851072 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.851018    2585 scope.go:117] "RemoveContainer" containerID="6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744"
Sep 17 18:05:15.851308 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:15.851291    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744\": container with ID starting with 6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744 not found: ID does not exist" containerID="6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744"
Sep 17 18:05:15.851346 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.851314    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744"} err="failed to get container status \"6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744\": rpc error: code = NotFound desc = could not find container \"6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744\": container with ID starting with 6d512977d999afe60e5c234b79d5f0c2e244b3cf374cae7d585938efdeb34744 not found: ID does not exist"
Sep 17 18:05:15.851346 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.851330    2585 scope.go:117] "RemoveContainer" containerID="f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355"
Sep 17 18:05:15.851558 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:15.851544    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355\": container with ID starting with f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355 not found: ID does not exist" containerID="f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355"
Sep 17 18:05:15.851605 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.851560    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355"} err="failed to get container status \"f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355\": rpc error: code = NotFound desc = could not find container \"f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355\": container with ID starting with f8b8d64fe4f92903cf1bb216c2782c7fc548c500163a25a12b3b02ee3c227355 not found: ID does not exist"
Sep 17 18:05:15.891387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.891357    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"]
Sep 17 18:05:15.903967 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:15.903943    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-gwk46"]
Sep 17 18:05:16.512150 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:16.512108    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:05:16.959919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:16.959890    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" path="/var/lib/kubelet/pods/386eb205-daf8-45b1-866a-b79e5bc50c72/volumes"
Sep 17 18:05:18.924752 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.924718    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"]
Sep 17 18:05:18.925135 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925097    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kserve-container"
Sep 17 18:05:18.925135 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925108    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kserve-container"
Sep 17 18:05:18.925135 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925118    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="storage-initializer"
Sep 17 18:05:18.925135 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925124    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="storage-initializer"
Sep 17 18:05:18.925321 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925138    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kube-rbac-proxy"
Sep 17 18:05:18.925321 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925146    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kube-rbac-proxy"
Sep 17 18:05:18.925321 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925209    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kube-rbac-proxy"
Sep 17 18:05:18.925321 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.925220    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="386eb205-daf8-45b1-866a-b79e5bc50c72" containerName="kserve-container"
Sep 17 18:05:18.928552 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.928530    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:18.931554 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.931534    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 18:05:18.932430 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.932407    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 18:05:18.944482 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.944458    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"]
Sep 17 18:05:18.977925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.977898    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbfb28fc-422f-4376-b53b-a8124a506969-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:18.978054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.977943    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:18.978054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.977972    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:18.978054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.977997    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:18.978054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:18.978022    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7kf\" (UniqueName: \"kubernetes.io/projected/bbfb28fc-422f-4376-b53b-a8124a506969-kube-api-access-tf7kf\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.078739 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.078690    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbfb28fc-422f-4376-b53b-a8124a506969-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.078922 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.078756    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.078922 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.078785    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.078922 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.078839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.078922 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.078868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7kf\" (UniqueName: \"kubernetes.io/projected/bbfb28fc-422f-4376-b53b-a8124a506969-kube-api-access-tf7kf\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.079152 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:19.078949    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 18:05:19.079152 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:19.079034    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls podName:bbfb28fc-422f-4376-b53b-a8124a506969 nodeName:}" failed. No retries permitted until 2026-09-17 18:05:19.579005364 +0000 UTC m=+1452.770343033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" (UID: "bbfb28fc-422f-4376-b53b-a8124a506969") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 18:05:19.079152 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.079139    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbfb28fc-422f-4376-b53b-a8124a506969-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.079455 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.079435    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.079495 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.079451    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.088611 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.088585    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7kf\" (UniqueName: \"kubernetes.io/projected/bbfb28fc-422f-4376-b53b-a8124a506969-kube-api-access-tf7kf\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.582927 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.582890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.585430 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.585408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:19.838856 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:19.838741    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:20.115885 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:20.115857    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"]
Sep 17 18:05:20.116215 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:05:20.116181    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfb28fc_422f_4376_b53b_a8124a506969.slice/crio-231b3c9c84af0058641e9d9ef6bedc3eb09f2408632345f6f6e422369b1b9723 WatchSource:0}: Error finding container 231b3c9c84af0058641e9d9ef6bedc3eb09f2408632345f6f6e422369b1b9723: Status 404 returned error can't find the container with id 231b3c9c84af0058641e9d9ef6bedc3eb09f2408632345f6f6e422369b1b9723
Sep 17 18:05:20.851718 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:20.851684    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerStarted","Data":"87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219"}
Sep 17 18:05:20.851904 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:20.851739    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerStarted","Data":"231b3c9c84af0058641e9d9ef6bedc3eb09f2408632345f6f6e422369b1b9723"}
Sep 17 18:05:26.511981 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:26.511949    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:05:32.925163 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:32.925131    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"]
Sep 17 18:05:32.925602 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:32.925361    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" containerID="cri-o://bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131" gracePeriod=30
Sep 17 18:05:32.925602 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:32.925420    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kube-rbac-proxy" containerID="cri-o://14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709" gracePeriod=30
Sep 17 18:05:33.914624 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:33.914594    2585 generic.go:356] "Generic (PLEG): container finished" podID="c7966fb4-0e58-406c-9c60-979326232271" containerID="14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709" exitCode=2
Sep 17 18:05:33.914787 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:33.914661    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerDied","Data":"14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709"}
Sep 17 18:05:36.506945 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:36.506901    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:05:36.512268 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:36.512244    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:05:37.166492 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.166459    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:05:37.222621 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.222593    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-cabundle-cert\") pod \"c7966fb4-0e58-406c-9c60-979326232271\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") "
Sep 17 18:05:37.222794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.222628    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9zm9\" (UniqueName: \"kubernetes.io/projected/c7966fb4-0e58-406c-9c60-979326232271-kube-api-access-g9zm9\") pod \"c7966fb4-0e58-406c-9c60-979326232271\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") "
Sep 17 18:05:37.222794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.222675    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls\") pod \"c7966fb4-0e58-406c-9c60-979326232271\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") "
Sep 17 18:05:37.222794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.222787    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7966fb4-0e58-406c-9c60-979326232271-kserve-provision-location\") pod \"c7966fb4-0e58-406c-9c60-979326232271\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") "
Sep 17 18:05:37.223017 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.222839    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-kube-rbac-proxy-sar-config\") pod \"c7966fb4-0e58-406c-9c60-979326232271\" (UID: \"c7966fb4-0e58-406c-9c60-979326232271\") "
Sep 17 18:05:37.223017 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.222990    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7966fb4-0e58-406c-9c60-979326232271" (UID: "c7966fb4-0e58-406c-9c60-979326232271"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:37.223129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.223067    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7966fb4-0e58-406c-9c60-979326232271-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7966fb4-0e58-406c-9c60-979326232271" (UID: "c7966fb4-0e58-406c-9c60-979326232271"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:37.223239 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.223150    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:37.223239 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.223217    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7966fb4-0e58-406c-9c60-979326232271" (UID: "c7966fb4-0e58-406c-9c60-979326232271"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:37.224877 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.224854    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7966fb4-0e58-406c-9c60-979326232271-kube-api-access-g9zm9" (OuterVolumeSpecName: "kube-api-access-g9zm9") pod "c7966fb4-0e58-406c-9c60-979326232271" (UID: "c7966fb4-0e58-406c-9c60-979326232271"). InnerVolumeSpecName "kube-api-access-g9zm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:37.224974 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.224856    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7966fb4-0e58-406c-9c60-979326232271" (UID: "c7966fb4-0e58-406c-9c60-979326232271"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:37.324275 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.324201    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7966fb4-0e58-406c-9c60-979326232271-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:37.324275 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.324227    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7966fb4-0e58-406c-9c60-979326232271-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:37.324275 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.324237    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7966fb4-0e58-406c-9c60-979326232271-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:37.324275 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.324246    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zm9\" (UniqueName: \"kubernetes.io/projected/c7966fb4-0e58-406c-9c60-979326232271-kube-api-access-g9zm9\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:05:37.935039 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.935000    2585 generic.go:356] "Generic (PLEG): container finished" podID="c7966fb4-0e58-406c-9c60-979326232271" containerID="bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131" exitCode=0
Sep 17 18:05:37.935402 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.935066    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"
Sep 17 18:05:37.935402 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.935064    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerDied","Data":"bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131"}
Sep 17 18:05:37.935402 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.935203    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt" event={"ID":"c7966fb4-0e58-406c-9c60-979326232271","Type":"ContainerDied","Data":"29b340d4f2d55b060bbb602823c6689a4a9624d9563c0ca14a645fc224b6a1ee"}
Sep 17 18:05:37.935402 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.935242    2585 scope.go:117] "RemoveContainer" containerID="14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709"
Sep 17 18:05:37.944565 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.944550    2585 scope.go:117] "RemoveContainer" containerID="bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131"
Sep 17 18:05:37.952442 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.952424    2585 scope.go:117] "RemoveContainer" containerID="a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d"
Sep 17 18:05:37.960563 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.960545    2585 scope.go:117] "RemoveContainer" containerID="14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709"
Sep 17 18:05:37.960855 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:37.960835    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709\": container with ID starting with 14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709 not found: ID does not exist" containerID="14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709"
Sep 17 18:05:37.960925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.960863    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709"} err="failed to get container status \"14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709\": rpc error: code = NotFound desc = could not find container \"14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709\": container with ID starting with 14734533690b3a5f60ac47b7099ecbfb0930047c0ad62d1a3bdd38b940b7a709 not found: ID does not exist"
Sep 17 18:05:37.960925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.960879    2585 scope.go:117] "RemoveContainer" containerID="bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131"
Sep 17 18:05:37.961191 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:37.961168    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131\": container with ID starting with bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131 not found: ID does not exist" containerID="bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131"
Sep 17 18:05:37.961240 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.961200    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131"} err="failed to get container status \"bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131\": rpc error: code = NotFound desc = could not find container \"bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131\": container with ID starting with bd35b43c009efbf5f9c194ce7e480c0570ea0f02c2e2d027fbb195d0de742131 not found: ID does not exist"
Sep 17 18:05:37.961240 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.961221    2585 scope.go:117] "RemoveContainer" containerID="a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d"
Sep 17 18:05:37.961486 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:05:37.961466    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d\": container with ID starting with a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d not found: ID does not exist" containerID="a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d"
Sep 17 18:05:37.961541 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.961490    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d"} err="failed to get container status \"a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d\": rpc error: code = NotFound desc = could not find container \"a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d\": container with ID starting with a986118d69e7748e332673cd16255052bd3dcf4076e1634b186a293f26e5e77d not found: ID does not exist"
Sep 17 18:05:37.961880 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.961863    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"]
Sep 17 18:05:37.971898 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:37.971879    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-cdfjt"]
Sep 17 18:05:38.959274 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:38.959233    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7966fb4-0e58-406c-9c60-979326232271" path="/var/lib/kubelet/pods/c7966fb4-0e58-406c-9c60-979326232271/volumes"
Sep 17 18:05:50.996004 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:50.995970    2585 generic.go:356] "Generic (PLEG): container finished" podID="bbfb28fc-422f-4376-b53b-a8124a506969" containerID="87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219" exitCode=0
Sep 17 18:05:50.996406 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:50.996042    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerDied","Data":"87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219"}
Sep 17 18:05:52.002951 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:52.002917    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerStarted","Data":"3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014"}
Sep 17 18:05:52.003386 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:52.002977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerStarted","Data":"53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a"}
Sep 17 18:05:52.003386 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:52.003165    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:53.007739 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:53.007708    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:59.018770 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:59.018745    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:05:59.040844 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:05:59.040766    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" podStartSLOduration=10.892486765 podStartE2EDuration="41.040747693s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.148260928s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:52.033166638 +0000 UTC m=+1485.224504325" watchObservedRunningTime="2026-09-17 18:05:59.040747693 +0000 UTC m=+1492.232085379"
Sep 17 18:06:09.016076 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:06:09.016042    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:06:09.019708 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:06:09.019684    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:07:28.990262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:28.990234    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:07:39.999984 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:39.999938    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:07:45.738533 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:45.738500    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"]
Sep 17 18:07:45.739041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:45.738763    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kserve-container" containerID="cri-o://53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a" gracePeriod=30
Sep 17 18:07:45.739041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:45.738790    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kube-rbac-proxy" containerID="cri-o://3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014" gracePeriod=30
Sep 17 18:07:46.518940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:46.518899    2585 generic.go:356] "Generic (PLEG): container finished" podID="bbfb28fc-422f-4376-b53b-a8124a506969" containerID="3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014" exitCode=2
Sep 17 18:07:46.519121 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:46.518977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerDied","Data":"3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014"}
Sep 17 18:07:47.085026 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.085004    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:07:47.174195 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174164    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf7kf\" (UniqueName: \"kubernetes.io/projected/bbfb28fc-422f-4376-b53b-a8124a506969-kube-api-access-tf7kf\") pod \"bbfb28fc-422f-4376-b53b-a8124a506969\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") "
Sep 17 18:07:47.174387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174217    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-cabundle-cert\") pod \"bbfb28fc-422f-4376-b53b-a8124a506969\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") "
Sep 17 18:07:47.174387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174241    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-kube-rbac-proxy-sar-config\") pod \"bbfb28fc-422f-4376-b53b-a8124a506969\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") "
Sep 17 18:07:47.174387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174290    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbfb28fc-422f-4376-b53b-a8124a506969-kserve-provision-location\") pod \"bbfb28fc-422f-4376-b53b-a8124a506969\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") "
Sep 17 18:07:47.174387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174328    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls\") pod \"bbfb28fc-422f-4376-b53b-a8124a506969\" (UID: \"bbfb28fc-422f-4376-b53b-a8124a506969\") "
Sep 17 18:07:47.174672 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174646    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbfb28fc-422f-4376-b53b-a8124a506969" (UID: "bbfb28fc-422f-4376-b53b-a8124a506969"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:47.174987 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.174963    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbfb28fc-422f-4376-b53b-a8124a506969" (UID: "bbfb28fc-422f-4376-b53b-a8124a506969"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:47.176612 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.176590    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfb28fc-422f-4376-b53b-a8124a506969-kube-api-access-tf7kf" (OuterVolumeSpecName: "kube-api-access-tf7kf") pod "bbfb28fc-422f-4376-b53b-a8124a506969" (UID: "bbfb28fc-422f-4376-b53b-a8124a506969"). InnerVolumeSpecName "kube-api-access-tf7kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:47.176716 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.176645    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbfb28fc-422f-4376-b53b-a8124a506969" (UID: "bbfb28fc-422f-4376-b53b-a8124a506969"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:47.179078 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.179055    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfb28fc-422f-4376-b53b-a8124a506969-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbfb28fc-422f-4376-b53b-a8124a506969" (UID: "bbfb28fc-422f-4376-b53b-a8124a506969"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:47.275705 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.275680    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbfb28fc-422f-4376-b53b-a8124a506969-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:07:47.275705 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.275702    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbfb28fc-422f-4376-b53b-a8124a506969-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:07:47.275873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.275712    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf7kf\" (UniqueName: \"kubernetes.io/projected/bbfb28fc-422f-4376-b53b-a8124a506969-kube-api-access-tf7kf\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:07:47.275873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.275720    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:07:47.275873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.275728    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbfb28fc-422f-4376-b53b-a8124a506969-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:07:47.525721 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.525640    2585 generic.go:356] "Generic (PLEG): container finished" podID="bbfb28fc-422f-4376-b53b-a8124a506969" containerID="53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a" exitCode=0
Sep 17 18:07:47.525721 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.525691    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"
Sep 17 18:07:47.525949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.525729    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerDied","Data":"53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a"}
Sep 17 18:07:47.525949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.525804    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl" event={"ID":"bbfb28fc-422f-4376-b53b-a8124a506969","Type":"ContainerDied","Data":"231b3c9c84af0058641e9d9ef6bedc3eb09f2408632345f6f6e422369b1b9723"}
Sep 17 18:07:47.525949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.525847    2585 scope.go:117] "RemoveContainer" containerID="3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014"
Sep 17 18:07:47.534552 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.534535    2585 scope.go:117] "RemoveContainer" containerID="53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a"
Sep 17 18:07:47.542091 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.542072    2585 scope.go:117] "RemoveContainer" containerID="87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219"
Sep 17 18:07:47.549497 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.549474    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"]
Sep 17 18:07:47.550727 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.550713    2585 scope.go:117] "RemoveContainer" containerID="3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014"
Sep 17 18:07:47.550990 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:07:47.550974    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014\": container with ID starting with 3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014 not found: ID does not exist" containerID="3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014"
Sep 17 18:07:47.551048 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.550998    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014"} err="failed to get container status \"3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014\": rpc error: code = NotFound desc = could not find container \"3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014\": container with ID starting with 3a7d0e0372f873ad1b60314eaa5186ad691fa2fa30724913cc7d55f963c69014 not found: ID does not exist"
Sep 17 18:07:47.551048 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.551013    2585 scope.go:117] "RemoveContainer" containerID="53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a"
Sep 17 18:07:47.551229 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:07:47.551210    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a\": container with ID starting with 53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a not found: ID does not exist" containerID="53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a"
Sep 17 18:07:47.551271 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.551238    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a"} err="failed to get container status \"53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a\": rpc error: code = NotFound desc = could not find container \"53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a\": container with ID starting with 53e880305355d59255cb86278b93e46598d29c5acd9dd2b19b5026735fc9164a not found: ID does not exist"
Sep 17 18:07:47.551271 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.551254    2585 scope.go:117] "RemoveContainer" containerID="87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219"
Sep 17 18:07:47.551502 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:07:47.551485    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219\": container with ID starting with 87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219 not found: ID does not exist" containerID="87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219"
Sep 17 18:07:47.551576 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.551506    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219"} err="failed to get container status \"87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219\": rpc error: code = NotFound desc = could not find container \"87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219\": container with ID starting with 87660dc17904b6028104aad68cfb14a9ae75134689890cee7836fcff99fe7219 not found: ID does not exist"
Sep 17 18:07:47.554301 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:47.554282    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-stwhl"]
Sep 17 18:07:48.960331 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:48.960260    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" path="/var/lib/kubelet/pods/bbfb28fc-422f-4376-b53b-a8124a506969/volumes"
Sep 17 18:07:51.044409 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044376    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"]
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044710    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kube-rbac-proxy"
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044721    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kube-rbac-proxy"
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044735    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="storage-initializer"
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044740    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="storage-initializer"
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044747    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="storage-initializer"
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044752    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="storage-initializer"
Sep 17 18:07:51.044758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044761    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044767    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044774    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kserve-container"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044779    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kserve-container"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044795    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kube-rbac-proxy"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044800    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kube-rbac-proxy"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044889    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kserve-container"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044899    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kserve-container"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044908    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7966fb4-0e58-406c-9c60-979326232271" containerName="kube-rbac-proxy"
Sep 17 18:07:51.045043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.044914    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfb28fc-422f-4376-b53b-a8124a506969" containerName="kube-rbac-proxy"
Sep 17 18:07:51.049784 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.049764    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.053540 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.053521    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:07:51.053648 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.053582    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:07:51.054128 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.054105    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:07:51.054236 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.054142    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:07:51.054616 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.054598    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 18:07:51.054695 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.054616    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:07:51.054695 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.054618    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:07:51.064263 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.064244    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"]
Sep 17 18:07:51.209248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.209214    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qvf\" (UniqueName: \"kubernetes.io/projected/721ae53e-24ea-4865-b468-749045823001-kube-api-access-p9qvf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.209387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.209260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/721ae53e-24ea-4865-b468-749045823001-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.209387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.209284    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.209387 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.209306    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.209497 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.209412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/721ae53e-24ea-4865-b468-749045823001-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.310643 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.310586    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qvf\" (UniqueName: \"kubernetes.io/projected/721ae53e-24ea-4865-b468-749045823001-kube-api-access-p9qvf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.310643 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.310627    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/721ae53e-24ea-4865-b468-749045823001-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.310788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.310645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.310788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.310674    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.310788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.310735    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/721ae53e-24ea-4865-b468-749045823001-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.311137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.311114    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/721ae53e-24ea-4865-b468-749045823001-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.311331 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.311309    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.311403 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.311361    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.313316 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.313297    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/721ae53e-24ea-4865-b468-749045823001-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.319910 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.319887    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qvf\" (UniqueName: \"kubernetes.io/projected/721ae53e-24ea-4865-b468-749045823001-kube-api-access-p9qvf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.360409 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.360382    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:51.636555 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:51.636530    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"]
Sep 17 18:07:51.638693 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:07:51.638672    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721ae53e_24ea_4865_b468_749045823001.slice/crio-f74e08ad0064537da40840c92d780a9d8b183e1dd3a4cab9ba599aa49dbd2f17 WatchSource:0}: Error finding container f74e08ad0064537da40840c92d780a9d8b183e1dd3a4cab9ba599aa49dbd2f17: Status 404 returned error can't find the container with id f74e08ad0064537da40840c92d780a9d8b183e1dd3a4cab9ba599aa49dbd2f17
Sep 17 18:07:52.553363 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:52.553331    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerStarted","Data":"31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85"}
Sep 17 18:07:52.553797 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:52.553386    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerStarted","Data":"f74e08ad0064537da40840c92d780a9d8b183e1dd3a4cab9ba599aa49dbd2f17"}
Sep 17 18:07:56.572516 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:56.572486    2585 generic.go:356] "Generic (PLEG): container finished" podID="721ae53e-24ea-4865-b468-749045823001" containerID="31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85" exitCode=0
Sep 17 18:07:56.572845 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:56.572560    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerDied","Data":"31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85"}
Sep 17 18:07:57.579928 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:57.579887    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerStarted","Data":"4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088"}
Sep 17 18:07:57.580285 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:57.579980    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerStarted","Data":"ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7"}
Sep 17 18:07:57.580285 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:57.580218    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:07:57.604093 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:57.604044    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" podStartSLOduration=1.873644685 podStartE2EDuration="6.604029037s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.730384352s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:57.602126231 +0000 UTC m=+1610.793463917" watchObservedRunningTime="2026-09-17 18:07:57.604029037 +0000 UTC m=+1610.795366723"
Sep 17 18:07:58.585245 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:07:58.585215    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:08:04.595516 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:04.595486    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:08:53.418004 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.417972    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"]
Sep 17 18:08:53.421919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.421894    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.424734 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.424713    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:08:53.424886 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.424824    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 18:08:53.434030 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.434004    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"]
Sep 17 18:08:53.609501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.609463    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.609501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.609503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32deebb4-c614-45b7-a154-f885bb5d546d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.609758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.609534    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32deebb4-c614-45b7-a154-f885bb5d546d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.609758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.609660    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cf75\" (UniqueName: \"kubernetes.io/projected/32deebb4-c614-45b7-a154-f885bb5d546d-kube-api-access-9cf75\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.609758 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.609699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710291 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710205    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710291 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710243    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32deebb4-c614-45b7-a154-f885bb5d546d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710291 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32deebb4-c614-45b7-a154-f885bb5d546d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710329    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cf75\" (UniqueName: \"kubernetes.io/projected/32deebb4-c614-45b7-a154-f885bb5d546d-kube-api-access-9cf75\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710350    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710692 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710664    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32deebb4-c614-45b7-a154-f885bb5d546d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.710986 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.710965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.711080 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.711010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.712886 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.712868    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32deebb4-c614-45b7-a154-f885bb5d546d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.721681 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.721662    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cf75\" (UniqueName: \"kubernetes.io/projected/32deebb4-c614-45b7-a154-f885bb5d546d-kube-api-access-9cf75\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:53.732696 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:53.732673    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:08:54.008258 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:54.008064    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"]
Sep 17 18:08:54.014103 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:08:54.014017    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32deebb4_c614_45b7_a154_f885bb5d546d.slice/crio-45577057752ee05324bc8d58651179fcaf4146d5aee35dd85c0d7c1519703be5 WatchSource:0}: Error finding container 45577057752ee05324bc8d58651179fcaf4146d5aee35dd85c0d7c1519703be5: Status 404 returned error can't find the container with id 45577057752ee05324bc8d58651179fcaf4146d5aee35dd85c0d7c1519703be5
Sep 17 18:08:54.843460 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:54.843418    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerStarted","Data":"2d93a731b12de86c8a299b8cb0b878a7efdd648687087618da385705ea9a556e"}
Sep 17 18:08:54.843865 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:54.843489    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerStarted","Data":"45577057752ee05324bc8d58651179fcaf4146d5aee35dd85c0d7c1519703be5"}
Sep 17 18:08:59.870671 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:59.870638    2585 generic.go:356] "Generic (PLEG): container finished" podID="32deebb4-c614-45b7-a154-f885bb5d546d" containerID="2d93a731b12de86c8a299b8cb0b878a7efdd648687087618da385705ea9a556e" exitCode=0
Sep 17 18:08:59.871035 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:08:59.870712    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerDied","Data":"2d93a731b12de86c8a299b8cb0b878a7efdd648687087618da385705ea9a556e"}
Sep 17 18:09:00.878231 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:00.878196    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerStarted","Data":"083691700cd74df9076455a8c37208c91dbbb5bbdd9fd411cf2c0359db0e17c8"}
Sep 17 18:09:00.878709 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:00.878268    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerStarted","Data":"784f029202783b003ea301f70c1b19d557fca40a43e139ba30bcbdae76b45fe9"}
Sep 17 18:09:00.878709 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:00.878503    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:09:00.902205 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:00.902161    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" podStartSLOduration=2.273521268 podStartE2EDuration="7.902146768s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.6286255s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:00.90001361 +0000 UTC m=+1674.091351294" watchObservedRunningTime="2026-09-17 18:09:00.902146768 +0000 UTC m=+1674.093484452"
Sep 17 18:09:01.884140 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:01.884107    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:09:01.885569 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:01.885541    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:09:02.895144 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:02.895097    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:09:07.899512 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:07.899474    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:09:07.900024 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:07.899948    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:09:17.900389 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:17.900360    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:09:25.311629 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:25.311588    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"]
Sep 17 18:09:25.312027 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:25.311857    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container" containerID="cri-o://784f029202783b003ea301f70c1b19d557fca40a43e139ba30bcbdae76b45fe9" gracePeriod=30
Sep 17 18:09:25.312027 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:25.311913    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kube-rbac-proxy" containerID="cri-o://083691700cd74df9076455a8c37208c91dbbb5bbdd9fd411cf2c0359db0e17c8" gracePeriod=30
Sep 17 18:09:26.014484 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.014450    2585 generic.go:356] "Generic (PLEG): container finished" podID="32deebb4-c614-45b7-a154-f885bb5d546d" containerID="083691700cd74df9076455a8c37208c91dbbb5bbdd9fd411cf2c0359db0e17c8" exitCode=2
Sep 17 18:09:26.014484 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.014483    2585 generic.go:356] "Generic (PLEG): container finished" podID="32deebb4-c614-45b7-a154-f885bb5d546d" containerID="784f029202783b003ea301f70c1b19d557fca40a43e139ba30bcbdae76b45fe9" exitCode=0
Sep 17 18:09:26.014666 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.014516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerDied","Data":"083691700cd74df9076455a8c37208c91dbbb5bbdd9fd411cf2c0359db0e17c8"}
Sep 17 18:09:26.014666 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.014572    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerDied","Data":"784f029202783b003ea301f70c1b19d557fca40a43e139ba30bcbdae76b45fe9"}
Sep 17 18:09:26.052709 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.052687    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:09:26.195054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.194977    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-cabundle-cert\") pod \"32deebb4-c614-45b7-a154-f885bb5d546d\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") "
Sep 17 18:09:26.195054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.195021    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32deebb4-c614-45b7-a154-f885bb5d546d-kserve-provision-location\") pod \"32deebb4-c614-45b7-a154-f885bb5d546d\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") "
Sep 17 18:09:26.195262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.195066    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32deebb4-c614-45b7-a154-f885bb5d546d-proxy-tls\") pod \"32deebb4-c614-45b7-a154-f885bb5d546d\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") "
Sep 17 18:09:26.195262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.195089    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-kube-rbac-proxy-sar-config\") pod \"32deebb4-c614-45b7-a154-f885bb5d546d\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") "
Sep 17 18:09:26.195262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.195137    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cf75\" (UniqueName: \"kubernetes.io/projected/32deebb4-c614-45b7-a154-f885bb5d546d-kube-api-access-9cf75\") pod \"32deebb4-c614-45b7-a154-f885bb5d546d\" (UID: \"32deebb4-c614-45b7-a154-f885bb5d546d\") "
Sep 17 18:09:26.195422 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.195320    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32deebb4-c614-45b7-a154-f885bb5d546d" (UID: "32deebb4-c614-45b7-a154-f885bb5d546d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:26.195516 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.195493    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32deebb4-c614-45b7-a154-f885bb5d546d" (UID: "32deebb4-c614-45b7-a154-f885bb5d546d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:26.197137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.197103    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32deebb4-c614-45b7-a154-f885bb5d546d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32deebb4-c614-45b7-a154-f885bb5d546d" (UID: "32deebb4-c614-45b7-a154-f885bb5d546d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:26.197227 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.197213    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32deebb4-c614-45b7-a154-f885bb5d546d-kube-api-access-9cf75" (OuterVolumeSpecName: "kube-api-access-9cf75") pod "32deebb4-c614-45b7-a154-f885bb5d546d" (UID: "32deebb4-c614-45b7-a154-f885bb5d546d"). InnerVolumeSpecName "kube-api-access-9cf75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:26.218567 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.218543    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32deebb4-c614-45b7-a154-f885bb5d546d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32deebb4-c614-45b7-a154-f885bb5d546d" (UID: "32deebb4-c614-45b7-a154-f885bb5d546d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:26.296608 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.296583    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cf75\" (UniqueName: \"kubernetes.io/projected/32deebb4-c614-45b7-a154-f885bb5d546d-kube-api-access-9cf75\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.296608 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.296605    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.296752 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.296615    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32deebb4-c614-45b7-a154-f885bb5d546d-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.296752 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.296624    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32deebb4-c614-45b7-a154-f885bb5d546d-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.296752 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.296632    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32deebb4-c614-45b7-a154-f885bb5d546d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.988668 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:26.988642    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:09:27.030428 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.030374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm" event={"ID":"32deebb4-c614-45b7-a154-f885bb5d546d","Type":"ContainerDied","Data":"45577057752ee05324bc8d58651179fcaf4146d5aee35dd85c0d7c1519703be5"}
Sep 17 18:09:27.030428 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.030408    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"
Sep 17 18:09:27.030658 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.030485    2585 scope.go:117] "RemoveContainer" containerID="083691700cd74df9076455a8c37208c91dbbb5bbdd9fd411cf2c0359db0e17c8"
Sep 17 18:09:27.040341 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.040319    2585 scope.go:117] "RemoveContainer" containerID="784f029202783b003ea301f70c1b19d557fca40a43e139ba30bcbdae76b45fe9"
Sep 17 18:09:27.049911 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.049502    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"]
Sep 17 18:09:27.049911 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.049617    2585 scope.go:117] "RemoveContainer" containerID="2d93a731b12de86c8a299b8cb0b878a7efdd648687087618da385705ea9a556e"
Sep 17 18:09:27.052490 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:27.052467    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-xfrmm"]
Sep 17 18:09:28.959828 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:28.959783    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" path="/var/lib/kubelet/pods/32deebb4-c614-45b7-a154-f885bb5d546d/volumes"
Sep 17 18:09:37.017491 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:37.017454    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"]
Sep 17 18:09:37.017986 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:37.017672    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kserve-container" containerID="cri-o://ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7" gracePeriod=30
Sep 17 18:09:37.017986 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:37.017766    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kube-rbac-proxy" containerID="cri-o://4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088" gracePeriod=30
Sep 17 18:09:38.084232 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.084198    2585 generic.go:356] "Generic (PLEG): container finished" podID="721ae53e-24ea-4865-b468-749045823001" containerID="4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088" exitCode=2
Sep 17 18:09:38.084596 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.084269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerDied","Data":"4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088"}
Sep 17 18:09:38.365209 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.365188    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:09:38.408584 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.408557    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-cabundle-cert\") pod \"721ae53e-24ea-4865-b468-749045823001\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") "
Sep 17 18:09:38.408720 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.408599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/721ae53e-24ea-4865-b468-749045823001-kserve-provision-location\") pod \"721ae53e-24ea-4865-b468-749045823001\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") "
Sep 17 18:09:38.408720 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.408646    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/721ae53e-24ea-4865-b468-749045823001-proxy-tls\") pod \"721ae53e-24ea-4865-b468-749045823001\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") "
Sep 17 18:09:38.408720 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.408667    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-kube-rbac-proxy-sar-config\") pod \"721ae53e-24ea-4865-b468-749045823001\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") "
Sep 17 18:09:38.408720 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.408693    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qvf\" (UniqueName: \"kubernetes.io/projected/721ae53e-24ea-4865-b468-749045823001-kube-api-access-p9qvf\") pod \"721ae53e-24ea-4865-b468-749045823001\" (UID: \"721ae53e-24ea-4865-b468-749045823001\") "
Sep 17 18:09:38.408982 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.408948    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "721ae53e-24ea-4865-b468-749045823001" (UID: "721ae53e-24ea-4865-b468-749045823001"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:38.409099 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.409067    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "721ae53e-24ea-4865-b468-749045823001" (UID: "721ae53e-24ea-4865-b468-749045823001"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:38.410895 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.410869    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721ae53e-24ea-4865-b468-749045823001-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "721ae53e-24ea-4865-b468-749045823001" (UID: "721ae53e-24ea-4865-b468-749045823001"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:38.411035 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.411019    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721ae53e-24ea-4865-b468-749045823001-kube-api-access-p9qvf" (OuterVolumeSpecName: "kube-api-access-p9qvf") pod "721ae53e-24ea-4865-b468-749045823001" (UID: "721ae53e-24ea-4865-b468-749045823001"). InnerVolumeSpecName "kube-api-access-p9qvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:38.413657 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.413637    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ae53e-24ea-4865-b468-749045823001-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "721ae53e-24ea-4865-b468-749045823001" (UID: "721ae53e-24ea-4865-b468-749045823001"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:38.509696 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.509672    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/721ae53e-24ea-4865-b468-749045823001-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:38.509696 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.509693    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/721ae53e-24ea-4865-b468-749045823001-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:38.509847 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.509703    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:38.509847 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.509711    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qvf\" (UniqueName: \"kubernetes.io/projected/721ae53e-24ea-4865-b468-749045823001-kube-api-access-p9qvf\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:38.509847 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:38.509719    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/721ae53e-24ea-4865-b468-749045823001-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:09:39.090743 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.090714    2585 generic.go:356] "Generic (PLEG): container finished" podID="721ae53e-24ea-4865-b468-749045823001" containerID="ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7" exitCode=0
Sep 17 18:09:39.091120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.090777    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"
Sep 17 18:09:39.091120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.090799    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerDied","Data":"ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7"}
Sep 17 18:09:39.091120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.090874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6" event={"ID":"721ae53e-24ea-4865-b468-749045823001","Type":"ContainerDied","Data":"f74e08ad0064537da40840c92d780a9d8b183e1dd3a4cab9ba599aa49dbd2f17"}
Sep 17 18:09:39.091120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.090899    2585 scope.go:117] "RemoveContainer" containerID="4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088"
Sep 17 18:09:39.099540 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.099506    2585 scope.go:117] "RemoveContainer" containerID="ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7"
Sep 17 18:09:39.107019 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.107002    2585 scope.go:117] "RemoveContainer" containerID="31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85"
Sep 17 18:09:39.112744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.112724    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"]
Sep 17 18:09:39.114237 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.114215    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z2bs6"]
Sep 17 18:09:39.115473 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.115452    2585 scope.go:117] "RemoveContainer" containerID="4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088"
Sep 17 18:09:39.115752 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:09:39.115733    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088\": container with ID starting with 4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088 not found: ID does not exist" containerID="4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088"
Sep 17 18:09:39.115799 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.115760    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088"} err="failed to get container status \"4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088\": rpc error: code = NotFound desc = could not find container \"4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088\": container with ID starting with 4c6b5e86edb320f1aa14c260f65b1e5903510b673b25230ead1434eb57e5a088 not found: ID does not exist"
Sep 17 18:09:39.115799 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.115777    2585 scope.go:117] "RemoveContainer" containerID="ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7"
Sep 17 18:09:39.116071 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:09:39.116054    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7\": container with ID starting with ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7 not found: ID does not exist" containerID="ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7"
Sep 17 18:09:39.116111 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.116077    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7"} err="failed to get container status \"ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7\": rpc error: code = NotFound desc = could not find container \"ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7\": container with ID starting with ed765c8b2e4d0382e40abc2bfff68d0fb8ffae52b9bde90c74f14320dc5f26c7 not found: ID does not exist"
Sep 17 18:09:39.116111 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.116092    2585 scope.go:117] "RemoveContainer" containerID="31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85"
Sep 17 18:09:39.116325 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:09:39.116309    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85\": container with ID starting with 31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85 not found: ID does not exist" containerID="31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85"
Sep 17 18:09:39.116374 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:39.116330    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85"} err="failed to get container status \"31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85\": rpc error: code = NotFound desc = could not find container \"31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85\": container with ID starting with 31c420b364a732cf41038c6c70f48de77501f5da669442290065ea81fd952f85 not found: ID does not exist"
Sep 17 18:09:40.959766 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:40.959738    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="721ae53e-24ea-4865-b468-749045823001" path="/var/lib/kubelet/pods/721ae53e-24ea-4865-b468-749045823001/volumes"
Sep 17 18:09:42.171939 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.171903    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"]
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172252    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="storage-initializer"
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172263    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="storage-initializer"
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172270    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container"
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172275    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container"
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172292    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="storage-initializer"
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172297    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="storage-initializer"
Sep 17 18:09:42.172308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172307    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kserve-container"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172313    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kserve-container"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172323    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kube-rbac-proxy"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172328    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kube-rbac-proxy"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172336    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kube-rbac-proxy"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172341    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kube-rbac-proxy"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172407    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kserve-container"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172414    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kube-rbac-proxy"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172421    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="721ae53e-24ea-4865-b468-749045823001" containerName="kube-rbac-proxy"
Sep 17 18:09:42.172518 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.172427    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="32deebb4-c614-45b7-a154-f885bb5d546d" containerName="kserve-container"
Sep 17 18:09:42.175954 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.175931    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.178751 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.178728    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:09:42.178889 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.178732    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 18:09:42.178889 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.178734    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:09:42.179010 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.178966    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:09:42.180046 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.180021    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:09:42.180148 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.180025    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:09:42.180148 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.180094    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:09:42.189274 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.189255    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"]
Sep 17 18:09:42.241585 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.241557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.241704 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.241621    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pn4\" (UniqueName: \"kubernetes.io/projected/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-api-access-k9pn4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.241704 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.241654    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.241704 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.241678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.241704 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.241704    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/593ba088-ab41-43f3-a3d0-eee8f190ec26-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.342601 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.342567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.342601 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.342603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.342804 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.342640    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/593ba088-ab41-43f3-a3d0-eee8f190ec26-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.342804 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.342685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.342804 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:09:42.342719    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 18:09:42.342804 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.342768    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pn4\" (UniqueName: \"kubernetes.io/projected/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-api-access-k9pn4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.342804 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:09:42.342793    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls podName:593ba088-ab41-43f3-a3d0-eee8f190ec26 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:42.84277128 +0000 UTC m=+1716.034108960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" (UID: "593ba088-ab41-43f3-a3d0-eee8f190ec26") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 18:09:42.343216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.343193    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/593ba088-ab41-43f3-a3d0-eee8f190ec26-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.343302 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.343282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.343370 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.343355    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.351598 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.351569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pn4\" (UniqueName: \"kubernetes.io/projected/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-api-access-k9pn4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.847063 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.847017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:42.849466 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:42.849438    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:43.087280 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:43.087248    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:43.359178 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:43.359154    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"]
Sep 17 18:09:43.364341 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:09:43.364309    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593ba088_ab41_43f3_a3d0_eee8f190ec26.slice/crio-213e1dcb982a9afcf15d169107aafb998e67179f4ecf81e2594b17a095d2f105 WatchSource:0}: Error finding container 213e1dcb982a9afcf15d169107aafb998e67179f4ecf81e2594b17a095d2f105: Status 404 returned error can't find the container with id 213e1dcb982a9afcf15d169107aafb998e67179f4ecf81e2594b17a095d2f105
Sep 17 18:09:44.118037 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:44.118001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerStarted","Data":"00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4"}
Sep 17 18:09:44.118208 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:44.118093    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerStarted","Data":"213e1dcb982a9afcf15d169107aafb998e67179f4ecf81e2594b17a095d2f105"}
Sep 17 18:09:53.161018 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:53.160987    2585 generic.go:356] "Generic (PLEG): container finished" podID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerID="00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4" exitCode=0
Sep 17 18:09:53.161421 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:53.161065    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerDied","Data":"00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4"}
Sep 17 18:09:54.168878 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:54.168844    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerStarted","Data":"248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778"}
Sep 17 18:09:54.169279 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:54.168904    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerStarted","Data":"e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781"}
Sep 17 18:09:54.169279 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:54.169107    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:55.174842 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:55.174784    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:09:55.176112 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:55.176085    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:09:56.179047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:09:56.179014    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:10:01.183301 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:01.183274    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:10:01.183761 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:01.183729    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:10:01.207615 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:01.207570    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" podStartSLOduration=10.350016289 podStartE2EDuration="19.207556483s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.857540194s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:54.208569652 +0000 UTC m=+1727.399907336" watchObservedRunningTime="2026-09-17 18:10:01.207556483 +0000 UTC m=+1734.398894167"
Sep 17 18:10:11.185031 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:11.185000    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:10:24.449357 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:24.449326    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"]
Sep 17 18:10:24.449874 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:24.449584    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container" containerID="cri-o://e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781" gracePeriod=30
Sep 17 18:10:24.449874 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:24.449622    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kube-rbac-proxy" containerID="cri-o://248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778" gracePeriod=30
Sep 17 18:10:25.326051 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:25.326019    2585 generic.go:356] "Generic (PLEG): container finished" podID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerID="248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778" exitCode=2
Sep 17 18:10:25.326222 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:25.326096    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerDied","Data":"248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778"}
Sep 17 18:10:26.094237 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.094215    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:10:26.204117 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9pn4\" (UniqueName: \"kubernetes.io/projected/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-api-access-k9pn4\") pod \"593ba088-ab41-43f3-a3d0-eee8f190ec26\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") "
Sep 17 18:10:26.204117 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204095    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-cabundle-cert\") pod \"593ba088-ab41-43f3-a3d0-eee8f190ec26\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") "
Sep 17 18:10:26.204333 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204157    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-rbac-proxy-sar-config\") pod \"593ba088-ab41-43f3-a3d0-eee8f190ec26\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") "
Sep 17 18:10:26.204333 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204175    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/593ba088-ab41-43f3-a3d0-eee8f190ec26-kserve-provision-location\") pod \"593ba088-ab41-43f3-a3d0-eee8f190ec26\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") "
Sep 17 18:10:26.204333 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204204    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls\") pod \"593ba088-ab41-43f3-a3d0-eee8f190ec26\" (UID: \"593ba088-ab41-43f3-a3d0-eee8f190ec26\") "
Sep 17 18:10:26.204513 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204490    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "593ba088-ab41-43f3-a3d0-eee8f190ec26" (UID: "593ba088-ab41-43f3-a3d0-eee8f190ec26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:26.204571 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.204517    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "593ba088-ab41-43f3-a3d0-eee8f190ec26" (UID: "593ba088-ab41-43f3-a3d0-eee8f190ec26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:26.205298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.205272    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593ba088-ab41-43f3-a3d0-eee8f190ec26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "593ba088-ab41-43f3-a3d0-eee8f190ec26" (UID: "593ba088-ab41-43f3-a3d0-eee8f190ec26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:26.206353 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.206331    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "593ba088-ab41-43f3-a3d0-eee8f190ec26" (UID: "593ba088-ab41-43f3-a3d0-eee8f190ec26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:26.206453 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.206415    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-api-access-k9pn4" (OuterVolumeSpecName: "kube-api-access-k9pn4") pod "593ba088-ab41-43f3-a3d0-eee8f190ec26" (UID: "593ba088-ab41-43f3-a3d0-eee8f190ec26"). InnerVolumeSpecName "kube-api-access-k9pn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:26.305558 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.305525    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:10:26.305558 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.305557    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/593ba088-ab41-43f3-a3d0-eee8f190ec26-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:10:26.305558 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.305567    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/593ba088-ab41-43f3-a3d0-eee8f190ec26-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:10:26.305730 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.305578    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pn4\" (UniqueName: \"kubernetes.io/projected/593ba088-ab41-43f3-a3d0-eee8f190ec26-kube-api-access-k9pn4\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:10:26.305730 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.305587    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/593ba088-ab41-43f3-a3d0-eee8f190ec26-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:10:26.333053 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.333028    2585 generic.go:356] "Generic (PLEG): container finished" podID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerID="e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781" exitCode=0
Sep 17 18:10:26.333157 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.333086    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"
Sep 17 18:10:26.333157 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.333107    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerDied","Data":"e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781"}
Sep 17 18:10:26.333233 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.333171    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s" event={"ID":"593ba088-ab41-43f3-a3d0-eee8f190ec26","Type":"ContainerDied","Data":"213e1dcb982a9afcf15d169107aafb998e67179f4ecf81e2594b17a095d2f105"}
Sep 17 18:10:26.333233 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.333197    2585 scope.go:117] "RemoveContainer" containerID="248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778"
Sep 17 18:10:26.341992 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.341976    2585 scope.go:117] "RemoveContainer" containerID="e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781"
Sep 17 18:10:26.349192 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.349177    2585 scope.go:117] "RemoveContainer" containerID="00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4"
Sep 17 18:10:26.356663 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.356643    2585 scope.go:117] "RemoveContainer" containerID="248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778"
Sep 17 18:10:26.356716 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.356643    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"]
Sep 17 18:10:26.356939 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:10:26.356922    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778\": container with ID starting with 248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778 not found: ID does not exist" containerID="248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778"
Sep 17 18:10:26.356991 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.356946    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778"} err="failed to get container status \"248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778\": rpc error: code = NotFound desc = could not find container \"248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778\": container with ID starting with 248ece689c826279210775c02196730b72925c7bf14b9248926021e37783b778 not found: ID does not exist"
Sep 17 18:10:26.356991 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.356961    2585 scope.go:117] "RemoveContainer" containerID="e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781"
Sep 17 18:10:26.357169 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:10:26.357151    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781\": container with ID starting with e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781 not found: ID does not exist" containerID="e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781"
Sep 17 18:10:26.357233 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.357179    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781"} err="failed to get container status \"e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781\": rpc error: code = NotFound desc = could not find container \"e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781\": container with ID starting with e20328d01116bd52c74652ed17c67600018dd0a38c4507bc9ee8a31259111781 not found: ID does not exist"
Sep 17 18:10:26.357233 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.357202    2585 scope.go:117] "RemoveContainer" containerID="00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4"
Sep 17 18:10:26.357435 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:10:26.357416    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4\": container with ID starting with 00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4 not found: ID does not exist" containerID="00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4"
Sep 17 18:10:26.357473 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.357439    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4"} err="failed to get container status \"00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4\": rpc error: code = NotFound desc = could not find container \"00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4\": container with ID starting with 00a6e61051f6607572bf38a908be3b0741acb960e301074f0cf38fe95db2f0a4 not found: ID does not exist"
Sep 17 18:10:26.363431 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.363411    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mpk7s"]
Sep 17 18:10:26.959350 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:26.959316    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" path="/var/lib/kubelet/pods/593ba088-ab41-43f3-a3d0-eee8f190ec26/volumes"
Sep 17 18:10:29.668552 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.668513    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"]
Sep 17 18:10:29.669078 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669063    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kube-rbac-proxy"
Sep 17 18:10:29.669129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669082    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kube-rbac-proxy"
Sep 17 18:10:29.669129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669100    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container"
Sep 17 18:10:29.669129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669108    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container"
Sep 17 18:10:29.669129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669126    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="storage-initializer"
Sep 17 18:10:29.669251 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669135    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="storage-initializer"
Sep 17 18:10:29.669251 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669220    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kserve-container"
Sep 17 18:10:29.669251 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.669231    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="593ba088-ab41-43f3-a3d0-eee8f190ec26" containerName="kube-rbac-proxy"
Sep 17 18:10:29.672885 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.672862    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.675861 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.675697    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:10:29.675861 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.675716    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:10:29.675861 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.675725    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:10:29.675861 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.675716    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 18:10:29.676164 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.675933    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:10:29.676164 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.676094    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:10:29.677016 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.676995    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:10:29.682110 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.682090    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"]
Sep 17 18:10:29.831042 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.831001    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41cfee24-6425-48e5-ae5a-1eea4f425efe-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.831042 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.831059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.831298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.831080    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41cfee24-6425-48e5-ae5a-1eea4f425efe-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.831298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.831118    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.831298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.831174    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc684\" (UniqueName: \"kubernetes.io/projected/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-api-access-wc684\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932060 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.931982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc684\" (UniqueName: \"kubernetes.io/projected/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-api-access-wc684\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932060 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932040    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41cfee24-6425-48e5-ae5a-1eea4f425efe-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932060 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932071    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41cfee24-6425-48e5-ae5a-1eea4f425efe-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932205    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932485 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932438    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41cfee24-6425-48e5-ae5a-1eea4f425efe-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932764 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.932844 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.932749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.934586 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.934560    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41cfee24-6425-48e5-ae5a-1eea4f425efe-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.941202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.941181    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc684\" (UniqueName: \"kubernetes.io/projected/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-api-access-wc684\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:29.985502 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:29.985477    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:30.261977 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:30.261952    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"]
Sep 17 18:10:30.263969 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:10:30.263940    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cfee24_6425_48e5_ae5a_1eea4f425efe.slice/crio-af5bba60e3b0baa0ed810c223dc46159cbcde27c3b8e490897d2625023b92535 WatchSource:0}: Error finding container af5bba60e3b0baa0ed810c223dc46159cbcde27c3b8e490897d2625023b92535: Status 404 returned error can't find the container with id af5bba60e3b0baa0ed810c223dc46159cbcde27c3b8e490897d2625023b92535
Sep 17 18:10:30.360761 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:30.360723    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerStarted","Data":"c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0"}
Sep 17 18:10:30.360906 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:30.360793    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerStarted","Data":"af5bba60e3b0baa0ed810c223dc46159cbcde27c3b8e490897d2625023b92535"}
Sep 17 18:10:35.385805 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.385765    2585 generic.go:356] "Generic (PLEG): container finished" podID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerID="c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0" exitCode=0
Sep 17 18:10:35.386287 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.385841    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerDied","Data":"c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0"}
Sep 17 18:10:35.770756 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.770669    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"]
Sep 17 18:10:35.774358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.774334    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.779913 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.779892    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 18:10:35.780356 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.780334    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:10:35.812205 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.812171    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"]
Sep 17 18:10:35.875313 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.875282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvvq\" (UniqueName: \"kubernetes.io/projected/557be521-7a4f-469a-b1de-dcdd595a4415-kube-api-access-lwvvq\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.875510 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.875350    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.875510 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.875385    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/557be521-7a4f-469a-b1de-dcdd595a4415-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.875510 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.875410    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/557be521-7a4f-469a-b1de-dcdd595a4415-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.875694 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.875525    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976222 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976181    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976222 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976224    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/557be521-7a4f-469a-b1de-dcdd595a4415-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976541 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/557be521-7a4f-469a-b1de-dcdd595a4415-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976541 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976541 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976326    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvvq\" (UniqueName: \"kubernetes.io/projected/557be521-7a4f-469a-b1de-dcdd595a4415-kube-api-access-lwvvq\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976753 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976718    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/557be521-7a4f-469a-b1de-dcdd595a4415-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.976984 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.976964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.977077 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.977051    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.978950 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.978930    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/557be521-7a4f-469a-b1de-dcdd595a4415-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:35.988014 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:35.987986    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvvq\" (UniqueName: \"kubernetes.io/projected/557be521-7a4f-469a-b1de-dcdd595a4415-kube-api-access-lwvvq\") pod \"isvc-sklearn-v2-predictor-555d868545-r5nwp\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:36.084682 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.084595    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:36.370358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.370264    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"]
Sep 17 18:10:36.372856 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:10:36.372805    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557be521_7a4f_469a_b1de_dcdd595a4415.slice/crio-d6427c30433ea8752eb48e137c90d01bf2834f7ff355076a7a1da92225820917 WatchSource:0}: Error finding container d6427c30433ea8752eb48e137c90d01bf2834f7ff355076a7a1da92225820917: Status 404 returned error can't find the container with id d6427c30433ea8752eb48e137c90d01bf2834f7ff355076a7a1da92225820917
Sep 17 18:10:36.395433 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.395406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerStarted","Data":"d6427c30433ea8752eb48e137c90d01bf2834f7ff355076a7a1da92225820917"}
Sep 17 18:10:36.397879 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.397857    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerStarted","Data":"0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece"}
Sep 17 18:10:36.397965 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.397902    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerStarted","Data":"32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8"}
Sep 17 18:10:36.398168 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.398149    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:36.398216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.398204    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:36.399623 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.399600    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:10:36.420496 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:36.420447    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" podStartSLOduration=2.9611913960000003 podStartE2EDuration="7.420429832s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.459238436s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:36.419962539 +0000 UTC m=+1769.611300249" watchObservedRunningTime="2026-09-17 18:10:36.420429832 +0000 UTC m=+1769.611767517"
Sep 17 18:10:37.404343 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:37.404301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerStarted","Data":"a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d"}
Sep 17 18:10:37.404803 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:37.404680    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:10:40.422053 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:40.422017    2585 generic.go:356] "Generic (PLEG): container finished" podID="557be521-7a4f-469a-b1de-dcdd595a4415" containerID="a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d" exitCode=0
Sep 17 18:10:40.422531 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:40.422083    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerDied","Data":"a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d"}
Sep 17 18:10:41.429821 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:41.429767    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerStarted","Data":"72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072"}
Sep 17 18:10:41.430330 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:41.429853    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerStarted","Data":"4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419"}
Sep 17 18:10:41.430330 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:41.430114    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:41.430330 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:41.430169    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:41.431757 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:41.431728    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:10:41.451298 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:41.451248    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podStartSLOduration=2.937837798 podStartE2EDuration="6.451234295s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.513396497s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:41.449466356 +0000 UTC m=+1774.640804044" watchObservedRunningTime="2026-09-17 18:10:41.451234295 +0000 UTC m=+1774.642571981"
Sep 17 18:10:42.411168 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:42.411136    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:42.411632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:42.411603    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:10:42.435570 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:42.435532    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:10:47.439457 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:47.439427    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:10:47.439914 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:47.439887    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:10:52.412658 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:52.412580    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:10:57.440218 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:10:57.440180    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:11:04.961112 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:04.961080    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"]
Sep 17 18:11:04.961569 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:04.961406    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container" containerID="cri-o://32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8" gracePeriod=30
Sep 17 18:11:04.961569 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:04.961418    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kube-rbac-proxy" containerID="cri-o://0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece" gracePeriod=30
Sep 17 18:11:05.547683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:05.547630    2585 generic.go:356] "Generic (PLEG): container finished" podID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerID="0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece" exitCode=2
Sep 17 18:11:05.547683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:05.547681    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerDied","Data":"0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece"}
Sep 17 18:11:06.309896 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.309872    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:11:06.425377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425301    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc684\" (UniqueName: \"kubernetes.io/projected/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-api-access-wc684\") pod \"41cfee24-6425-48e5-ae5a-1eea4f425efe\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") "
Sep 17 18:11:06.425377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425361    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41cfee24-6425-48e5-ae5a-1eea4f425efe-kserve-provision-location\") pod \"41cfee24-6425-48e5-ae5a-1eea4f425efe\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") "
Sep 17 18:11:06.425588 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425388    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41cfee24-6425-48e5-ae5a-1eea4f425efe-proxy-tls\") pod \"41cfee24-6425-48e5-ae5a-1eea4f425efe\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") "
Sep 17 18:11:06.425588 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425415    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-rbac-proxy-sar-config\") pod \"41cfee24-6425-48e5-ae5a-1eea4f425efe\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") "
Sep 17 18:11:06.425588 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425487    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-cabundle-cert\") pod \"41cfee24-6425-48e5-ae5a-1eea4f425efe\" (UID: \"41cfee24-6425-48e5-ae5a-1eea4f425efe\") "
Sep 17 18:11:06.425852 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425803    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41cfee24-6425-48e5-ae5a-1eea4f425efe" (UID: "41cfee24-6425-48e5-ae5a-1eea4f425efe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:06.425915 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.425866    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41cfee24-6425-48e5-ae5a-1eea4f425efe" (UID: "41cfee24-6425-48e5-ae5a-1eea4f425efe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:06.426386 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.426357    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cfee24-6425-48e5-ae5a-1eea4f425efe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41cfee24-6425-48e5-ae5a-1eea4f425efe" (UID: "41cfee24-6425-48e5-ae5a-1eea4f425efe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:06.427690 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.427666    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cfee24-6425-48e5-ae5a-1eea4f425efe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41cfee24-6425-48e5-ae5a-1eea4f425efe" (UID: "41cfee24-6425-48e5-ae5a-1eea4f425efe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:06.427777 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.427710    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-api-access-wc684" (OuterVolumeSpecName: "kube-api-access-wc684") pod "41cfee24-6425-48e5-ae5a-1eea4f425efe" (UID: "41cfee24-6425-48e5-ae5a-1eea4f425efe"). InnerVolumeSpecName "kube-api-access-wc684". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:06.526749 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.526727    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.526749 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.526747    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc684\" (UniqueName: \"kubernetes.io/projected/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-api-access-wc684\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.526912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.526757    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41cfee24-6425-48e5-ae5a-1eea4f425efe-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.526912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.526766    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41cfee24-6425-48e5-ae5a-1eea4f425efe-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.526912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.526774    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41cfee24-6425-48e5-ae5a-1eea4f425efe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.554750 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.554725    2585 generic.go:356] "Generic (PLEG): container finished" podID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerID="32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8" exitCode=0
Sep 17 18:11:06.554867 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.554765    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerDied","Data":"32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8"}
Sep 17 18:11:06.554867 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.554776    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"
Sep 17 18:11:06.554867 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.554835    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl" event={"ID":"41cfee24-6425-48e5-ae5a-1eea4f425efe","Type":"ContainerDied","Data":"af5bba60e3b0baa0ed810c223dc46159cbcde27c3b8e490897d2625023b92535"}
Sep 17 18:11:06.554867 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.554861    2585 scope.go:117] "RemoveContainer" containerID="0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece"
Sep 17 18:11:06.564260 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.564245    2585 scope.go:117] "RemoveContainer" containerID="32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8"
Sep 17 18:11:06.571926 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.571905    2585 scope.go:117] "RemoveContainer" containerID="c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0"
Sep 17 18:11:06.580287 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.580202    2585 scope.go:117] "RemoveContainer" containerID="0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece"
Sep 17 18:11:06.580583 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:11:06.580552    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece\": container with ID starting with 0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece not found: ID does not exist" containerID="0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece"
Sep 17 18:11:06.580653 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.580595    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece"} err="failed to get container status \"0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece\": rpc error: code = NotFound desc = could not find container \"0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece\": container with ID starting with 0b998f78d990a71d1ff3b19ec7b95e91d809ef7e983546e30ecb65e38063fece not found: ID does not exist"
Sep 17 18:11:06.580653 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.580617    2585 scope.go:117] "RemoveContainer" containerID="32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8"
Sep 17 18:11:06.580949 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:11:06.580927    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8\": container with ID starting with 32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8 not found: ID does not exist" containerID="32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8"
Sep 17 18:11:06.581016 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.580957    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8"} err="failed to get container status \"32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8\": rpc error: code = NotFound desc = could not find container \"32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8\": container with ID starting with 32e1fe5c502de1fccf0778dac2fa87475ad7fe0cf5e16fd35b43bc87edd541f8 not found: ID does not exist"
Sep 17 18:11:06.581016 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.580978    2585 scope.go:117] "RemoveContainer" containerID="c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0"
Sep 17 18:11:06.581256 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:11:06.581238    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0\": container with ID starting with c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0 not found: ID does not exist" containerID="c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0"
Sep 17 18:11:06.581318 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.581262    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0"} err="failed to get container status \"c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0\": rpc error: code = NotFound desc = could not find container \"c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0\": container with ID starting with c6a01196259c393eaab5d603d98ce21f7386ba1130c577812922d9d4d8827da0 not found: ID does not exist"
Sep 17 18:11:06.582546 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.582529    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"]
Sep 17 18:11:06.585353 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.585333    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j62pl"]
Sep 17 18:11:06.959343 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:06.959311    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" path="/var/lib/kubelet/pods/41cfee24-6425-48e5-ae5a-1eea4f425efe/volumes"
Sep 17 18:11:07.440249 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:07.440214    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:11:09.051379 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:09.051349    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:11:09.055445 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:09.055423    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:11:10.146238 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146203    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"]
Sep 17 18:11:10.146763 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146747    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container"
Sep 17 18:11:10.146825 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146767    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container"
Sep 17 18:11:10.146866 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146835    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="storage-initializer"
Sep 17 18:11:10.146866 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146846    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="storage-initializer"
Sep 17 18:11:10.146866 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146864    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kube-rbac-proxy"
Sep 17 18:11:10.146964 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146873    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kube-rbac-proxy"
Sep 17 18:11:10.146997 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146962    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kserve-container"
Sep 17 18:11:10.146997 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.146977    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="41cfee24-6425-48e5-ae5a-1eea4f425efe" containerName="kube-rbac-proxy"
Sep 17 18:11:10.152398 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.152378    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.155386 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.155362    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:11:10.155641 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.155625    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:11:10.162512 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.162488    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"]
Sep 17 18:11:10.192449 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.192415    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"]
Sep 17 18:11:10.196035 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.196016    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.198872 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.198855    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:11:10.209463 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.209441    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"]
Sep 17 18:11:10.256330 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256305    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.256462 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256336    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllf4\" (UniqueName: \"kubernetes.io/projected/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-api-access-fllf4\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.256462 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256363    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.256462 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256432    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.256611 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256484    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ceeda7-a18b-47e7-b524-5a0b06ade857-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.256611 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256523    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad1905db-1fee-4bd3-b698-02d8b0b79184-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.256611 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjl5n\" (UniqueName: \"kubernetes.io/projected/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-api-access-vjl5n\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.256748 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256617    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ceeda7-a18b-47e7-b524-5a0b06ade857-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.256748 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256638    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.256748 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.256656    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.357462 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.357462 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ceeda7-a18b-47e7-b524-5a0b06ade857-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad1905db-1fee-4bd3-b698-02d8b0b79184-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjl5n\" (UniqueName: \"kubernetes.io/projected/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-api-access-vjl5n\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357530    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ceeda7-a18b-47e7-b524-5a0b06ade857-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:11:10.357547    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.357683 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:11:10.357647    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls podName:ad1905db-1fee-4bd3-b698-02d8b0b79184 nodeName:}" failed. No retries permitted until 2026-09-17 18:11:10.857626127 +0000 UTC m=+1804.048963807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls") pod "isvc-canary-create-predictor-5f845bd94d-c226n" (UID: "ad1905db-1fee-4bd3-b698-02d8b0b79184") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:11:10.358122 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357706    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.358122 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357740    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllf4\" (UniqueName: \"kubernetes.io/projected/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-api-access-fllf4\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.358122 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.358122 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.357952    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad1905db-1fee-4bd3-b698-02d8b0b79184-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.358122 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.358004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ceeda7-a18b-47e7-b524-5a0b06ade857-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.358480 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.358452    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.358610 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.358588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.358892 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.358873    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.358892 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.358883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.360894 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.360872    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ceeda7-a18b-47e7-b524-5a0b06ade857-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.367675 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.367653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjl5n\" (UniqueName: \"kubernetes.io/projected/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-api-access-vjl5n\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.367790 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.367655    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllf4\" (UniqueName: \"kubernetes.io/projected/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-api-access-fllf4\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.465202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.465175    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:10.744854 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.744720    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"]
Sep 17 18:11:10.747505 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:11:10.747476    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ceeda7_a18b_47e7_b524_5a0b06ade857.slice/crio-8464bd13b3ca23486c82a5b511b617ab72275d30cf9c680612c8d7e001d10c72 WatchSource:0}: Error finding container 8464bd13b3ca23486c82a5b511b617ab72275d30cf9c680612c8d7e001d10c72: Status 404 returned error can't find the container with id 8464bd13b3ca23486c82a5b511b617ab72275d30cf9c680612c8d7e001d10c72
Sep 17 18:11:10.862274 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.862233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:10.864684 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:10.864667    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-c226n\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:11.106363 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:11.106287    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:11.384696 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:11.384666    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"]
Sep 17 18:11:11.385533 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:11:11.385504    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1905db_1fee_4bd3_b698_02d8b0b79184.slice/crio-8b0f0fd088fe104a7e768787ba1c02741a27f2bdf429f2516770c78d05b9b445 WatchSource:0}: Error finding container 8b0f0fd088fe104a7e768787ba1c02741a27f2bdf429f2516770c78d05b9b445: Status 404 returned error can't find the container with id 8b0f0fd088fe104a7e768787ba1c02741a27f2bdf429f2516770c78d05b9b445
Sep 17 18:11:11.580227 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:11.580185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerStarted","Data":"ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554"}
Sep 17 18:11:11.580406 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:11.580265    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerStarted","Data":"8b0f0fd088fe104a7e768787ba1c02741a27f2bdf429f2516770c78d05b9b445"}
Sep 17 18:11:11.581863 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:11.581842    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerStarted","Data":"6698f3bd704ad1257630cbcd53aa9f61875b0a0a7935bb65c91795fa477ce1f5"}
Sep 17 18:11:11.581952 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:11.581885    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerStarted","Data":"8464bd13b3ca23486c82a5b511b617ab72275d30cf9c680612c8d7e001d10c72"}
Sep 17 18:11:15.610474 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:15.610371    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerID="ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554" exitCode=0
Sep 17 18:11:15.610474 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:15.610450    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerDied","Data":"ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554"}
Sep 17 18:11:15.612118 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:15.612098    2585 generic.go:356] "Generic (PLEG): container finished" podID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerID="6698f3bd704ad1257630cbcd53aa9f61875b0a0a7935bb65c91795fa477ce1f5" exitCode=0
Sep 17 18:11:15.612235 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:15.612127    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerDied","Data":"6698f3bd704ad1257630cbcd53aa9f61875b0a0a7935bb65c91795fa477ce1f5"}
Sep 17 18:11:16.619486 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.619447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerStarted","Data":"4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5"}
Sep 17 18:11:16.619941 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.619521    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerStarted","Data":"68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785"}
Sep 17 18:11:16.619941 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.619844    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:16.620069 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.619977    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:16.621266 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.621238    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:11:16.621791 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.621773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerStarted","Data":"fa8866135bae2982aa8177b3cf375ecf6ef7eced1a8ec5dae84ccfbac791c4eb"}
Sep 17 18:11:16.621873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.621839    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerStarted","Data":"2594f85c1e6a2599e0a8aae0953c50ecc71efd80d770f78818f51c65b3d244b3"}
Sep 17 18:11:16.622053 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.622039    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:16.641458 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.641416    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podStartSLOduration=2.7833016 podStartE2EDuration="6.641403565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.858101965s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:16.640398042 +0000 UTC m=+1809.831735752" watchObservedRunningTime="2026-09-17 18:11:16.641403565 +0000 UTC m=+1809.832741249"
Sep 17 18:11:16.661012 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:16.660972    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podStartSLOduration=2.708949269 podStartE2EDuration="6.660959206s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.952009937s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:16.660717863 +0000 UTC m=+1809.852055548" watchObservedRunningTime="2026-09-17 18:11:16.660959206 +0000 UTC m=+1809.852296891"
Sep 17 18:11:17.440215 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:17.440182    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:11:17.626628 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:17.626586    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:11:17.627041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:17.626650    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:17.627990 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:17.627963    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:11:18.635927 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:18.635882    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:11:22.630931 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:22.630897    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:11:22.631378 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:22.631349    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:11:23.640113 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:23.640087    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:11:23.640610 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:23.640579    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:11:27.440321 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:27.440281    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:11:32.632079 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:32.632040    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:11:33.640859 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:33.640791    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:11:37.440269 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:37.440232    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:11:42.631435 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:42.631396    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:11:43.640506 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:43.640469    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:11:47.439943 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:47.439905    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:11:52.631388 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:52.631342    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:11:53.641291 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:53.641252    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:11:57.440886 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:11:57.440857    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:12:02.631683 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:02.631647    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:12:03.640506 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:03.640468    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:12:05.968453 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:05.968422    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"]
Sep 17 18:12:05.968853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:05.968627    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" containerID="cri-o://4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419" gracePeriod=30
Sep 17 18:12:05.968853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:05.968684    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kube-rbac-proxy" containerID="cri-o://72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072" gracePeriod=30
Sep 17 18:12:06.868489 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:06.868455    2585 generic.go:356] "Generic (PLEG): container finished" podID="557be521-7a4f-469a-b1de-dcdd595a4415" containerID="72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072" exitCode=2
Sep 17 18:12:06.868660 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:06.868522    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerDied","Data":"72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072"}
Sep 17 18:12:07.436185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:07.436140    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 18:12:07.440020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:07.439995    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:12:10.516604 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.516579    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:12:10.656672 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.656644    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/557be521-7a4f-469a-b1de-dcdd595a4415-kserve-provision-location\") pod \"557be521-7a4f-469a-b1de-dcdd595a4415\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") "
Sep 17 18:12:10.656862 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.656682    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/557be521-7a4f-469a-b1de-dcdd595a4415-proxy-tls\") pod \"557be521-7a4f-469a-b1de-dcdd595a4415\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") "
Sep 17 18:12:10.656862 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.656709    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvvq\" (UniqueName: \"kubernetes.io/projected/557be521-7a4f-469a-b1de-dcdd595a4415-kube-api-access-lwvvq\") pod \"557be521-7a4f-469a-b1de-dcdd595a4415\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") "
Sep 17 18:12:10.656862 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.656733    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-cabundle-cert\") pod \"557be521-7a4f-469a-b1de-dcdd595a4415\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") "
Sep 17 18:12:10.656862 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.656762    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-kube-rbac-proxy-sar-config\") pod \"557be521-7a4f-469a-b1de-dcdd595a4415\" (UID: \"557be521-7a4f-469a-b1de-dcdd595a4415\") "
Sep 17 18:12:10.657098 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.657027    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557be521-7a4f-469a-b1de-dcdd595a4415-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "557be521-7a4f-469a-b1de-dcdd595a4415" (UID: "557be521-7a4f-469a-b1de-dcdd595a4415"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:10.657168 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.657136    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "557be521-7a4f-469a-b1de-dcdd595a4415" (UID: "557be521-7a4f-469a-b1de-dcdd595a4415"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:10.657220 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.657178    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "557be521-7a4f-469a-b1de-dcdd595a4415" (UID: "557be521-7a4f-469a-b1de-dcdd595a4415"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:10.658960 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.658936    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557be521-7a4f-469a-b1de-dcdd595a4415-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "557be521-7a4f-469a-b1de-dcdd595a4415" (UID: "557be521-7a4f-469a-b1de-dcdd595a4415"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:10.659063 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.658968    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557be521-7a4f-469a-b1de-dcdd595a4415-kube-api-access-lwvvq" (OuterVolumeSpecName: "kube-api-access-lwvvq") pod "557be521-7a4f-469a-b1de-dcdd595a4415" (UID: "557be521-7a4f-469a-b1de-dcdd595a4415"). InnerVolumeSpecName "kube-api-access-lwvvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:10.757784 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.757761    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/557be521-7a4f-469a-b1de-dcdd595a4415-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:10.757784 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.757784    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/557be521-7a4f-469a-b1de-dcdd595a4415-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:10.757948 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.757795    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvvq\" (UniqueName: \"kubernetes.io/projected/557be521-7a4f-469a-b1de-dcdd595a4415-kube-api-access-lwvvq\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:10.757948 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.757804    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:10.757948 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.757826    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/557be521-7a4f-469a-b1de-dcdd595a4415-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:10.889200 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.889168    2585 generic.go:356] "Generic (PLEG): container finished" podID="557be521-7a4f-469a-b1de-dcdd595a4415" containerID="4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419" exitCode=0
Sep 17 18:12:10.889350 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.889229    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"
Sep 17 18:12:10.889350 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.889261    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerDied","Data":"4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419"}
Sep 17 18:12:10.889467 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.889365    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp" event={"ID":"557be521-7a4f-469a-b1de-dcdd595a4415","Type":"ContainerDied","Data":"d6427c30433ea8752eb48e137c90d01bf2834f7ff355076a7a1da92225820917"}
Sep 17 18:12:10.889467 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.889404    2585 scope.go:117] "RemoveContainer" containerID="72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072"
Sep 17 18:12:10.898726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.898706    2585 scope.go:117] "RemoveContainer" containerID="4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419"
Sep 17 18:12:10.906438 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.906421    2585 scope.go:117] "RemoveContainer" containerID="a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d"
Sep 17 18:12:10.912152 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.912131    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"]
Sep 17 18:12:10.914902 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.914878    2585 scope.go:117] "RemoveContainer" containerID="72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072"
Sep 17 18:12:10.915189 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:10.915170    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072\": container with ID starting with 72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072 not found: ID does not exist" containerID="72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072"
Sep 17 18:12:10.915258 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.915196    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072"} err="failed to get container status \"72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072\": rpc error: code = NotFound desc = could not find container \"72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072\": container with ID starting with 72e115d6731193552ec1937c15c9746d60047e5762f705c1f2814b89d0be0072 not found: ID does not exist"
Sep 17 18:12:10.915258 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.915212    2585 scope.go:117] "RemoveContainer" containerID="4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419"
Sep 17 18:12:10.915463 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:10.915439    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419\": container with ID starting with 4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419 not found: ID does not exist" containerID="4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419"
Sep 17 18:12:10.915511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.915475    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419"} err="failed to get container status \"4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419\": rpc error: code = NotFound desc = could not find container \"4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419\": container with ID starting with 4d1f55d7fbd95aaf94eb49d371a320e31682a20dc1c93d5a9da5377694a74419 not found: ID does not exist"
Sep 17 18:12:10.915511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.915499    2585 scope.go:117] "RemoveContainer" containerID="a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d"
Sep 17 18:12:10.915762 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:10.915745    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d\": container with ID starting with a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d not found: ID does not exist" containerID="a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d"
Sep 17 18:12:10.915863 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.915766    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d"} err="failed to get container status \"a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d\": rpc error: code = NotFound desc = could not find container \"a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d\": container with ID starting with a9eaed209b106646d17c04970bde41e1ab113e22fcc2b11d22c3bea3bccc568d not found: ID does not exist"
Sep 17 18:12:10.916173 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.916156    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-r5nwp"]
Sep 17 18:12:10.959739 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:10.959715    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" path="/var/lib/kubelet/pods/557be521-7a4f-469a-b1de-dcdd595a4415/volumes"
Sep 17 18:12:11.111499 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.111470    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"]
Sep 17 18:12:11.112067 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112045    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container"
Sep 17 18:12:11.112067 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112068    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container"
Sep 17 18:12:11.112216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112076    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kube-rbac-proxy"
Sep 17 18:12:11.112216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112083    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kube-rbac-proxy"
Sep 17 18:12:11.112216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112094    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="storage-initializer"
Sep 17 18:12:11.112216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112100    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="storage-initializer"
Sep 17 18:12:11.112216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112167    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kserve-container"
Sep 17 18:12:11.112216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.112175    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="557be521-7a4f-469a-b1de-dcdd595a4415" containerName="kube-rbac-proxy"
Sep 17 18:12:11.116973 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.116955    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.119719 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.119697    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 18:12:11.119877 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.119703    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 18:12:11.127827 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.127793    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"]
Sep 17 18:12:11.261667 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.261591    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.261667 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.261641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g98z\" (UniqueName: \"kubernetes.io/projected/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-api-access-9g98z\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.261863 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.261688    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.261863 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.261773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.261863 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.261795    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363160 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363125    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363160 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g98z\" (UniqueName: \"kubernetes.io/projected/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-api-access-9g98z\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363397 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363397 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363212    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363397 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363397 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:11.363295    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 18:12:11.363397 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:11.363354    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls podName:0d7ae63e-b00a-458e-988f-7b7d50c7a27c nodeName:}" failed. No retries permitted until 2026-09-17 18:12:11.863332625 +0000 UTC m=+1865.054670291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" (UID: "0d7ae63e-b00a-458e-988f-7b7d50c7a27c") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 18:12:11.363734 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363719    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.363892 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.363876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.364050 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.364031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.372390 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.372367    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g98z\" (UniqueName: \"kubernetes.io/projected/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-api-access-9g98z\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.867619 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.867581    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:11.870036 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:11.870017    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-t586c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:12.028039 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:12.028006    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:12.299253 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:12.299072    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"]
Sep 17 18:12:12.301729 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:12:12.301699    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7ae63e_b00a_458e_988f_7b7d50c7a27c.slice/crio-6e0daac52dc5791162c2349f922abcece60c3a9501f64e720759375a6da4639f WatchSource:0}: Error finding container 6e0daac52dc5791162c2349f922abcece60c3a9501f64e720759375a6da4639f: Status 404 returned error can't find the container with id 6e0daac52dc5791162c2349f922abcece60c3a9501f64e720759375a6da4639f
Sep 17 18:12:12.631666 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:12.631624    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:12:12.901989 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:12.901908    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerStarted","Data":"03ebace68fe7357e2d53c5990385354cf99d9cee9f839625eb9501c93874227e"}
Sep 17 18:12:12.901989 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:12.901964    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerStarted","Data":"6e0daac52dc5791162c2349f922abcece60c3a9501f64e720759375a6da4639f"}
Sep 17 18:12:13.640525 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:13.640493    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:12:16.925200 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:16.925167    2585 generic.go:356] "Generic (PLEG): container finished" podID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerID="03ebace68fe7357e2d53c5990385354cf99d9cee9f839625eb9501c93874227e" exitCode=0
Sep 17 18:12:16.925200 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:16.925209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerDied","Data":"03ebace68fe7357e2d53c5990385354cf99d9cee9f839625eb9501c93874227e"}
Sep 17 18:12:17.932765 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:17.932728    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerStarted","Data":"bf73e587a31e6bd6b3d0102fe3ba6ae7e1f89376131a07d984fe4098eeed4685"}
Sep 17 18:12:17.933185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:17.932793    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerStarted","Data":"71d06f769cce524cda798ded039bdfb646712e3b6889c4212a1508bbc66ac00c"}
Sep 17 18:12:17.933185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:17.933098    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:17.933185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:17.933155    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:17.934315 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:17.934289    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:12:17.952676 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:17.952623    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podStartSLOduration=3.309439387 podStartE2EDuration="6.952606407s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.64316702s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:17.95213722 +0000 UTC m=+1871.143474908" watchObservedRunningTime="2026-09-17 18:12:17.952606407 +0000 UTC m=+1871.143944093"
Sep 17 18:12:18.938065 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:18.937988    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:12:22.631889 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:22.631848    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:12:23.641138 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:23.641095    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:12:23.943095 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:23.943016    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:12:23.943547 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:23.943522    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:12:32.632530 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:32.632498    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:12:33.640967 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:33.640937    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:12:33.943949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:33.943861    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:12:40.229837 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:40.229780    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"]
Sep 17 18:12:40.230410 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:40.230117    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" containerID="cri-o://68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785" gracePeriod=30
Sep 17 18:12:40.230490 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:40.230465    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kube-rbac-proxy" containerID="cri-o://4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5" gracePeriod=30
Sep 17 18:12:40.233326 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:40.233300    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"]
Sep 17 18:12:40.233606 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:40.233576    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" containerID="cri-o://2594f85c1e6a2599e0a8aae0953c50ecc71efd80d770f78818f51c65b3d244b3" gracePeriod=30
Sep 17 18:12:40.233766 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:40.233740    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kube-rbac-proxy" containerID="cri-o://fa8866135bae2982aa8177b3cf375ecf6ef7eced1a8ec5dae84ccfbac791c4eb" gracePeriod=30
Sep 17 18:12:41.041236 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:41.041197    2585 generic.go:356] "Generic (PLEG): container finished" podID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerID="fa8866135bae2982aa8177b3cf375ecf6ef7eced1a8ec5dae84ccfbac791c4eb" exitCode=2
Sep 17 18:12:41.041498 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:41.041271    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerDied","Data":"fa8866135bae2982aa8177b3cf375ecf6ef7eced1a8ec5dae84ccfbac791c4eb"}
Sep 17 18:12:41.043455 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:41.043432    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerID="4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5" exitCode=2
Sep 17 18:12:41.043565 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:41.043502    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerDied","Data":"4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5"}
Sep 17 18:12:42.627167 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:42.627128    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 18:12:42.631940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:42.631914    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:12:43.636233 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:43.636188    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 18:12:43.640573 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:43.640550    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:12:43.943918 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:43.943821    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:12:44.778419 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.778392    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:12:44.849961 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.849885    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-rbac-proxy-sar-config\") pod \"ad1905db-1fee-4bd3-b698-02d8b0b79184\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") "
Sep 17 18:12:44.849961 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.849928    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad1905db-1fee-4bd3-b698-02d8b0b79184-kserve-provision-location\") pod \"ad1905db-1fee-4bd3-b698-02d8b0b79184\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") "
Sep 17 18:12:44.850132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.849970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-cabundle-cert\") pod \"ad1905db-1fee-4bd3-b698-02d8b0b79184\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") "
Sep 17 18:12:44.850132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.849993    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls\") pod \"ad1905db-1fee-4bd3-b698-02d8b0b79184\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") "
Sep 17 18:12:44.850132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.850035    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fllf4\" (UniqueName: \"kubernetes.io/projected/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-api-access-fllf4\") pod \"ad1905db-1fee-4bd3-b698-02d8b0b79184\" (UID: \"ad1905db-1fee-4bd3-b698-02d8b0b79184\") "
Sep 17 18:12:44.850291 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.850263    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad1905db-1fee-4bd3-b698-02d8b0b79184" (UID: "ad1905db-1fee-4bd3-b698-02d8b0b79184"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:44.850332 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.850307    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1905db-1fee-4bd3-b698-02d8b0b79184-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad1905db-1fee-4bd3-b698-02d8b0b79184" (UID: "ad1905db-1fee-4bd3-b698-02d8b0b79184"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:44.850367 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.850341    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad1905db-1fee-4bd3-b698-02d8b0b79184" (UID: "ad1905db-1fee-4bd3-b698-02d8b0b79184"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:44.852243 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.852225    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-api-access-fllf4" (OuterVolumeSpecName: "kube-api-access-fllf4") pod "ad1905db-1fee-4bd3-b698-02d8b0b79184" (UID: "ad1905db-1fee-4bd3-b698-02d8b0b79184"). InnerVolumeSpecName "kube-api-access-fllf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:44.852401 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.852382    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad1905db-1fee-4bd3-b698-02d8b0b79184" (UID: "ad1905db-1fee-4bd3-b698-02d8b0b79184"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:44.951485 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.951432    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:44.951485 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.951454    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad1905db-1fee-4bd3-b698-02d8b0b79184-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:44.951485 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.951463    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fllf4\" (UniqueName: \"kubernetes.io/projected/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-api-access-fllf4\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:44.951485 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.951472    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad1905db-1fee-4bd3-b698-02d8b0b79184-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:44.951485 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:44.951481    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad1905db-1fee-4bd3-b698-02d8b0b79184-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:45.067179 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.067142    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerID="68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785" exitCode=0
Sep 17 18:12:45.067327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.067204    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"
Sep 17 18:12:45.067327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.067218    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerDied","Data":"68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785"}
Sep 17 18:12:45.067327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.067275    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n" event={"ID":"ad1905db-1fee-4bd3-b698-02d8b0b79184","Type":"ContainerDied","Data":"8b0f0fd088fe104a7e768787ba1c02741a27f2bdf429f2516770c78d05b9b445"}
Sep 17 18:12:45.067327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.067307    2585 scope.go:117] "RemoveContainer" containerID="4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5"
Sep 17 18:12:45.070394 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.070364    2585 generic.go:356] "Generic (PLEG): container finished" podID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerID="2594f85c1e6a2599e0a8aae0953c50ecc71efd80d770f78818f51c65b3d244b3" exitCode=0
Sep 17 18:12:45.070506 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.070446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerDied","Data":"2594f85c1e6a2599e0a8aae0953c50ecc71efd80d770f78818f51c65b3d244b3"}
Sep 17 18:12:45.073945 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.073926    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:12:45.076552 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.076537    2585 scope.go:117] "RemoveContainer" containerID="68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785"
Sep 17 18:12:45.084573 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.084556    2585 scope.go:117] "RemoveContainer" containerID="ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554"
Sep 17 18:12:45.084899 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.084880    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"]
Sep 17 18:12:45.090470 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.090445    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-c226n"]
Sep 17 18:12:45.093063 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.093029    2585 scope.go:117] "RemoveContainer" containerID="4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5"
Sep 17 18:12:45.093336 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.093319    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5\": container with ID starting with 4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5 not found: ID does not exist" containerID="4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5"
Sep 17 18:12:45.093400 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.093359    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5"} err="failed to get container status \"4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5\": rpc error: code = NotFound desc = could not find container \"4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5\": container with ID starting with 4193935f34d08202a9ba2be6b77407332aa13c6e2ee7265a8193e7d9f23dc3c5 not found: ID does not exist"
Sep 17 18:12:45.093400 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.093374    2585 scope.go:117] "RemoveContainer" containerID="68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785"
Sep 17 18:12:45.093649 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.093625    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785\": container with ID starting with 68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785 not found: ID does not exist" containerID="68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785"
Sep 17 18:12:45.093744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.093654    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785"} err="failed to get container status \"68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785\": rpc error: code = NotFound desc = could not find container \"68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785\": container with ID starting with 68104755874ff8f0bb62d837e0f0b59e29e1e48ad6caed4d314d1b6ad6199785 not found: ID does not exist"
Sep 17 18:12:45.093744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.093669    2585 scope.go:117] "RemoveContainer" containerID="ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554"
Sep 17 18:12:45.093976 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.093961    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554\": container with ID starting with ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554 not found: ID does not exist" containerID="ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554"
Sep 17 18:12:45.094023 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.093983    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554"} err="failed to get container status \"ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554\": rpc error: code = NotFound desc = could not find container \"ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554\": container with ID starting with ec3c57353790fc73c23abac0413e7fc8df57fd8043d2e3b74057bc029334d554 not found: ID does not exist"
Sep 17 18:12:45.153951 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.153918    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ceeda7-a18b-47e7-b524-5a0b06ade857-proxy-tls\") pod \"06ceeda7-a18b-47e7-b524-5a0b06ade857\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") "
Sep 17 18:12:45.154072 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.153956    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-cabundle-cert\") pod \"06ceeda7-a18b-47e7-b524-5a0b06ade857\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") "
Sep 17 18:12:45.154072 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.153987    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ceeda7-a18b-47e7-b524-5a0b06ade857-kserve-provision-location\") pod \"06ceeda7-a18b-47e7-b524-5a0b06ade857\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") "
Sep 17 18:12:45.154145 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.154095    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-rbac-proxy-sar-config\") pod \"06ceeda7-a18b-47e7-b524-5a0b06ade857\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") "
Sep 17 18:12:45.154192 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.154154    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjl5n\" (UniqueName: \"kubernetes.io/projected/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-api-access-vjl5n\") pod \"06ceeda7-a18b-47e7-b524-5a0b06ade857\" (UID: \"06ceeda7-a18b-47e7-b524-5a0b06ade857\") "
Sep 17 18:12:45.154326 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.154300    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ceeda7-a18b-47e7-b524-5a0b06ade857-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06ceeda7-a18b-47e7-b524-5a0b06ade857" (UID: "06ceeda7-a18b-47e7-b524-5a0b06ade857"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:45.154452 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.154400    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06ceeda7-a18b-47e7-b524-5a0b06ade857" (UID: "06ceeda7-a18b-47e7-b524-5a0b06ade857"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:45.154452 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.154422    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ceeda7-a18b-47e7-b524-5a0b06ade857-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:45.154452 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.154433    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06ceeda7-a18b-47e7-b524-5a0b06ade857" (UID: "06ceeda7-a18b-47e7-b524-5a0b06ade857"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:45.156214 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.156196    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ceeda7-a18b-47e7-b524-5a0b06ade857-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06ceeda7-a18b-47e7-b524-5a0b06ade857" (UID: "06ceeda7-a18b-47e7-b524-5a0b06ade857"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:45.156332 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.156315    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-api-access-vjl5n" (OuterVolumeSpecName: "kube-api-access-vjl5n") pod "06ceeda7-a18b-47e7-b524-5a0b06ade857" (UID: "06ceeda7-a18b-47e7-b524-5a0b06ade857"). InnerVolumeSpecName "kube-api-access-vjl5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:45.255483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.255454    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ceeda7-a18b-47e7-b524-5a0b06ade857-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:45.255483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.255478    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:45.255483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.255488    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:45.255747 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.255497    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjl5n\" (UniqueName: \"kubernetes.io/projected/06ceeda7-a18b-47e7-b524-5a0b06ade857-kube-api-access-vjl5n\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:12:45.352751 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.352708    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"]
Sep 17 18:12:45.353198 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353168    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="storage-initializer"
Sep 17 18:12:45.353198 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353187    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="storage-initializer"
Sep 17 18:12:45.353198 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353200    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353209    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353219    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="storage-initializer"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353224    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="storage-initializer"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353235    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kube-rbac-proxy"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353240    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kube-rbac-proxy"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353250    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353255    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353264    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kube-rbac-proxy"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353271    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kube-rbac-proxy"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353340    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kserve-container"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353349    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kserve-container"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353359    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" containerName="kube-rbac-proxy"
Sep 17 18:12:45.353426 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.353369    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" containerName="kube-rbac-proxy"
Sep 17 18:12:45.358030 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.358009    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.360610 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.360587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:12:45.360722 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.360633    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:12:45.367562 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.367541    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"]
Sep 17 18:12:45.389310 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.389282    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"]
Sep 17 18:12:45.393035 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.393015    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.395498 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.395475    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:12:45.403047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.403027    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"]
Sep 17 18:12:45.457590 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457505    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.457590 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43488c7d-449f-49c0-a744-3a5c139af29d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.457775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457592    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.457775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457616    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.457775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.457775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpt5\" (UniqueName: \"kubernetes.io/projected/9a050689-5d4e-4949-a174-7180db32af12-kube-api-access-7gpt5\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.457775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547hr\" (UniqueName: \"kubernetes.io/projected/43488c7d-449f-49c0-a744-3a5c139af29d-kube-api-access-547hr\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.457775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457756    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.457996 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457839    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.457996 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.457864    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a050689-5d4e-4949-a174-7180db32af12-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.558981 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.558946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.559170 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43488c7d-449f-49c0-a744-3a5c139af29d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.559170 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559125    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.559170 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.559351 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559200    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.559351 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpt5\" (UniqueName: \"kubernetes.io/projected/9a050689-5d4e-4949-a174-7180db32af12-kube-api-access-7gpt5\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.559351 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559259    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547hr\" (UniqueName: \"kubernetes.io/projected/43488c7d-449f-49c0-a744-3a5c139af29d-kube-api-access-547hr\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.559351 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.559581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559393    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43488c7d-449f-49c0-a744-3a5c139af29d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.559581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.559581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a050689-5d4e-4949-a174-7180db32af12-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.559880    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.559902    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.559946    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls podName:9a050689-5d4e-4949-a174-7180db32af12 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:46.059927763 +0000 UTC m=+1899.251265431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls") pod "isvc-canary-promote-predictor-567686cc76-x8jbw" (UID: "9a050689-5d4e-4949-a174-7180db32af12") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:12:45.559977    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls podName:43488c7d-449f-49c0-a744-3a5c139af29d nodeName:}" failed. No retries permitted until 2026-09-17 18:12:46.059959585 +0000 UTC m=+1899.251297251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls") pod "isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" (UID: "43488c7d-449f-49c0-a744-3a5c139af29d") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.559995    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.560104    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.560219    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a050689-5d4e-4949-a174-7180db32af12-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.560248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.560226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.560678 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.560484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:45.568966 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.568939    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpt5\" (UniqueName: \"kubernetes.io/projected/9a050689-5d4e-4949-a174-7180db32af12-kube-api-access-7gpt5\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:45.569149 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:45.569127    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-547hr\" (UniqueName: \"kubernetes.io/projected/43488c7d-449f-49c0-a744-3a5c139af29d-kube-api-access-547hr\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:46.065139 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.065106    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:46.065664 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.065183    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:46.067968 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.067943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:46.068035 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.067943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-x8jbw\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:46.078741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.078717    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j" event={"ID":"06ceeda7-a18b-47e7-b524-5a0b06ade857","Type":"ContainerDied","Data":"8464bd13b3ca23486c82a5b511b617ab72275d30cf9c680612c8d7e001d10c72"}
Sep 17 18:12:46.078866 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.078740    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"
Sep 17 18:12:46.078866 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.078792    2585 scope.go:117] "RemoveContainer" containerID="fa8866135bae2982aa8177b3cf375ecf6ef7eced1a8ec5dae84ccfbac791c4eb"
Sep 17 18:12:46.087199 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.087182    2585 scope.go:117] "RemoveContainer" containerID="2594f85c1e6a2599e0a8aae0953c50ecc71efd80d770f78818f51c65b3d244b3"
Sep 17 18:12:46.096198 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.096183    2585 scope.go:117] "RemoveContainer" containerID="6698f3bd704ad1257630cbcd53aa9f61875b0a0a7935bb65c91795fa477ce1f5"
Sep 17 18:12:46.102075 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.102033    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"]
Sep 17 18:12:46.105396 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.105370    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-7sh9j"]
Sep 17 18:12:46.270401 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.270367    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:46.306127 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.306096    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:46.544445 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.544418    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"]
Sep 17 18:12:46.546422 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:12:46.546397    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43488c7d_449f_49c0_a744_3a5c139af29d.slice/crio-55342bc8dd034b672bce550b66a50fccdd9519f92520d14032b27bd76bb02916 WatchSource:0}: Error finding container 55342bc8dd034b672bce550b66a50fccdd9519f92520d14032b27bd76bb02916: Status 404 returned error can't find the container with id 55342bc8dd034b672bce550b66a50fccdd9519f92520d14032b27bd76bb02916
Sep 17 18:12:46.583081 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.583059    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"]
Sep 17 18:12:46.585044 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:12:46.585017    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a050689_5d4e_4949_a174_7180db32af12.slice/crio-efaa1d98c45810b5f87e17306414f888cab0f33468741a955b0d52641fc66921 WatchSource:0}: Error finding container efaa1d98c45810b5f87e17306414f888cab0f33468741a955b0d52641fc66921: Status 404 returned error can't find the container with id efaa1d98c45810b5f87e17306414f888cab0f33468741a955b0d52641fc66921
Sep 17 18:12:46.960319 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.960279    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ceeda7-a18b-47e7-b524-5a0b06ade857" path="/var/lib/kubelet/pods/06ceeda7-a18b-47e7-b524-5a0b06ade857/volumes"
Sep 17 18:12:46.960770 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:46.960755    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad1905db-1fee-4bd3-b698-02d8b0b79184" path="/var/lib/kubelet/pods/ad1905db-1fee-4bd3-b698-02d8b0b79184/volumes"
Sep 17 18:12:47.086393 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:47.086351    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerStarted","Data":"59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e"}
Sep 17 18:12:47.086899 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:47.086422    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerStarted","Data":"efaa1d98c45810b5f87e17306414f888cab0f33468741a955b0d52641fc66921"}
Sep 17 18:12:47.089371 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:47.089338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerStarted","Data":"dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053"}
Sep 17 18:12:47.089493 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:47.089395    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerStarted","Data":"55342bc8dd034b672bce550b66a50fccdd9519f92520d14032b27bd76bb02916"}
Sep 17 18:12:51.109935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:51.109902    2585 generic.go:356] "Generic (PLEG): container finished" podID="9a050689-5d4e-4949-a174-7180db32af12" containerID="59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e" exitCode=0
Sep 17 18:12:51.110358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:51.109969    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerDied","Data":"59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e"}
Sep 17 18:12:51.111611 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:51.111592    2585 generic.go:356] "Generic (PLEG): container finished" podID="43488c7d-449f-49c0-a744-3a5c139af29d" containerID="dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053" exitCode=0
Sep 17 18:12:51.111682 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:51.111661    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerDied","Data":"dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053"}
Sep 17 18:12:52.118329 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.118285    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerStarted","Data":"94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166"}
Sep 17 18:12:52.118781 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.118359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerStarted","Data":"cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0"}
Sep 17 18:12:52.118781 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.118546    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:52.129932 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.129905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerStarted","Data":"2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5"}
Sep 17 18:12:52.130081 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.129988    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerStarted","Data":"23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546"}
Sep 17 18:12:52.130201 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.130182    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:52.130262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.130231    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:52.131858 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.131825    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:12:52.149132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.149093    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podStartSLOduration=3.464976411 podStartE2EDuration="7.149080243s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684103832s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:52.145969575 +0000 UTC m=+1905.337307447" watchObservedRunningTime="2026-09-17 18:12:52.149080243 +0000 UTC m=+1905.340417928"
Sep 17 18:12:52.169860 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:52.169800    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podStartSLOduration=3.255044158 podStartE2EDuration="7.169787811s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914743653s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:52.168926749 +0000 UTC m=+1905.360264435" watchObservedRunningTime="2026-09-17 18:12:52.169787811 +0000 UTC m=+1905.361125496"
Sep 17 18:12:53.135615 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:53.135574    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:12:53.135997 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:53.135586    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:53.136942 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:53.136916    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:12:53.943463 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:53.943428    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:12:54.140594 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:54.140550    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:12:58.139690 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:58.139660    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:12:58.140214 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:58.140187    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:12:59.144855 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:59.144805    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:12:59.145475 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:12:59.145449    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:13:03.944378 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:03.944340    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:13:08.140107 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:08.140070    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:13:09.146220 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:09.146190    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:13:13.944001 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:13.943960    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:13:18.140105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:18.140069    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:13:19.145824 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:19.145781    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:13:23.944319 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:23.944279    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:13:28.140544 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:28.140507    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:13:29.145718 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:29.145682    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:13:33.944861 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:33.944801    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:13:38.140748 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:38.140702    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:13:39.146030 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:39.145990    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:13:41.201090 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:41.201053    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"]
Sep 17 18:13:41.201551 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:41.201336    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" containerID="cri-o://71d06f769cce524cda798ded039bdfb646712e3b6889c4212a1508bbc66ac00c" gracePeriod=30
Sep 17 18:13:41.201551 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:41.201370    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kube-rbac-proxy" containerID="cri-o://bf73e587a31e6bd6b3d0102fe3ba6ae7e1f89376131a07d984fe4098eeed4685" gracePeriod=30
Sep 17 18:13:41.379126 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:41.379091    2585 generic.go:356] "Generic (PLEG): container finished" podID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerID="bf73e587a31e6bd6b3d0102fe3ba6ae7e1f89376131a07d984fe4098eeed4685" exitCode=2
Sep 17 18:13:41.379261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:41.379141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerDied","Data":"bf73e587a31e6bd6b3d0102fe3ba6ae7e1f89376131a07d984fe4098eeed4685"}
Sep 17 18:13:43.938253 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:43.938206    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 18:13:43.944216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:43.944186    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:13:46.392511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.392475    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"]
Sep 17 18:13:46.397523 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.397448    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.400887 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.400652    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 18:13:46.401071 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.401049    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 18:13:46.409713 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.409565    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"]
Sep 17 18:13:46.411940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.411546    2585 generic.go:356] "Generic (PLEG): container finished" podID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerID="71d06f769cce524cda798ded039bdfb646712e3b6889c4212a1508bbc66ac00c" exitCode=0
Sep 17 18:13:46.411940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.411619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerDied","Data":"71d06f769cce524cda798ded039bdfb646712e3b6889c4212a1508bbc66ac00c"}
Sep 17 18:13:46.465063 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.465039    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:13:46.479536 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.479516    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.479661 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.479559    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.479661 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.479588    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.479756 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.479663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfrq\" (UniqueName: \"kubernetes.io/projected/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-api-access-trfrq\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.479756 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.479692    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.580929 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.580845    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g98z\" (UniqueName: \"kubernetes.io/projected/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-api-access-9g98z\") pod \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") "
Sep 17 18:13:46.580929 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.580884    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kserve-provision-location\") pod \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") "
Sep 17 18:13:46.580929 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.580920    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-rbac-proxy-sar-config\") pod \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") "
Sep 17 18:13:46.581202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.580937    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls\") pod \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") "
Sep 17 18:13:46.581202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581001    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-cabundle-cert\") pod \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\" (UID: \"0d7ae63e-b00a-458e-988f-7b7d50c7a27c\") "
Sep 17 18:13:46.581202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.581202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581141    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.581202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581181    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.581464 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfrq\" (UniqueName: \"kubernetes.io/projected/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-api-access-trfrq\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.581464 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581230    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d7ae63e-b00a-458e-988f-7b7d50c7a27c" (UID: "0d7ae63e-b00a-458e-988f-7b7d50c7a27c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:46.581464 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581361    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.581464 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581446    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:13:46.581464 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581267    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d7ae63e-b00a-458e-988f-7b7d50c7a27c" (UID: "0d7ae63e-b00a-458e-988f-7b7d50c7a27c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:46.581732 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:13:46.581457    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:13:46.581732 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:13:46.581525    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls podName:2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5 nodeName:}" failed. No retries permitted until 2026-09-17 18:13:47.081504826 +0000 UTC m=+1960.272842491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-x68r2" (UID: "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:13:46.581978 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.581932    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d7ae63e-b00a-458e-988f-7b7d50c7a27c" (UID: "0d7ae63e-b00a-458e-988f-7b7d50c7a27c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:46.582130 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.582062    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.582236 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.582213    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.582335 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.582303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.583287 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.583267    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-api-access-9g98z" (OuterVolumeSpecName: "kube-api-access-9g98z") pod "0d7ae63e-b00a-458e-988f-7b7d50c7a27c" (UID: "0d7ae63e-b00a-458e-988f-7b7d50c7a27c"). InnerVolumeSpecName "kube-api-access-9g98z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:46.583408 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.583392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d7ae63e-b00a-458e-988f-7b7d50c7a27c" (UID: "0d7ae63e-b00a-458e-988f-7b7d50c7a27c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:46.590407 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.590378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfrq\" (UniqueName: \"kubernetes.io/projected/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-api-access-trfrq\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:46.682804 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.682772    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g98z\" (UniqueName: \"kubernetes.io/projected/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-api-access-9g98z\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:13:46.682981 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.682804    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:13:46.682981 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.682843    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:13:46.682981 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:46.682852    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ae63e-b00a-458e-988f-7b7d50c7a27c-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:13:47.086792 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.086751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:47.089311 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.089284    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-x68r2\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:47.315650 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.315573    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:47.418472 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.418448    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"
Sep 17 18:13:47.418876 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.418478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c" event={"ID":"0d7ae63e-b00a-458e-988f-7b7d50c7a27c","Type":"ContainerDied","Data":"6e0daac52dc5791162c2349f922abcece60c3a9501f64e720759375a6da4639f"}
Sep 17 18:13:47.418876 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.418549    2585 scope.go:117] "RemoveContainer" containerID="bf73e587a31e6bd6b3d0102fe3ba6ae7e1f89376131a07d984fe4098eeed4685"
Sep 17 18:13:47.426850 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.426803    2585 scope.go:117] "RemoveContainer" containerID="71d06f769cce524cda798ded039bdfb646712e3b6889c4212a1508bbc66ac00c"
Sep 17 18:13:47.434757 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.434740    2585 scope.go:117] "RemoveContainer" containerID="03ebace68fe7357e2d53c5990385354cf99d9cee9f839625eb9501c93874227e"
Sep 17 18:13:47.438373 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.438347    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"]
Sep 17 18:13:47.442788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.442767    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-t586c"]
Sep 17 18:13:47.603940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:47.603917    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"]
Sep 17 18:13:47.606636 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:13:47.606607    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb57aa1_ad4c_40ac_b8b1_9330d4492fa5.slice/crio-f8b252ac505d90a243f6b8708f2d1e603f4520db31b0fe512713b6be3876d539 WatchSource:0}: Error finding container f8b252ac505d90a243f6b8708f2d1e603f4520db31b0fe512713b6be3876d539: Status 404 returned error can't find the container with id f8b252ac505d90a243f6b8708f2d1e603f4520db31b0fe512713b6be3876d539
Sep 17 18:13:48.140637 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:48.140589    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:13:48.432101 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:48.432013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerStarted","Data":"80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b"}
Sep 17 18:13:48.432453 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:48.432118    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerStarted","Data":"f8b252ac505d90a243f6b8708f2d1e603f4520db31b0fe512713b6be3876d539"}
Sep 17 18:13:48.960064 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:48.960027    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" path="/var/lib/kubelet/pods/0d7ae63e-b00a-458e-988f-7b7d50c7a27c/volumes"
Sep 17 18:13:49.146056 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:49.146017    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:13:52.452023 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:52.451991    2585 generic.go:356] "Generic (PLEG): container finished" podID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerID="80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b" exitCode=0
Sep 17 18:13:52.452385 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:52.452065    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerDied","Data":"80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b"}
Sep 17 18:13:52.453278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:52.453264    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:13:56.482009 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:56.481977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerStarted","Data":"0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87"}
Sep 17 18:13:56.482369 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:56.482041    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerStarted","Data":"77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3"}
Sep 17 18:13:56.482369 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:56.482238    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:56.505243 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:56.505190    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podStartSLOduration=2.336820137 podStartE2EDuration="10.50517296s" totalImagesPullingTime="3.846352545s" totalInitContainerRuntime="4.322000278s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:56.503629654 +0000 UTC m=+1969.694967354" watchObservedRunningTime="2026-09-17 18:13:56.50517296 +0000 UTC m=+1969.696510645"
Sep 17 18:13:57.487054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:57.487022    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:13:57.488150 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:57.488120    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:13:58.141057 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:58.141019    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:13:58.492016 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:58.491914    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:13:59.145697 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:13:59.145659    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:14:03.496590 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:03.496555    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:14:03.497179 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:03.497150    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:14:08.140962 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:08.140928    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:14:09.145958 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:09.145925    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:14:13.497269 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:13.497236    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:14:15.438088 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:15.438050    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"]
Sep 17 18:14:15.438551 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:15.438313    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" containerID="cri-o://23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546" gracePeriod=30
Sep 17 18:14:15.438551 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:15.438347    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kube-rbac-proxy" containerID="cri-o://2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5" gracePeriod=30
Sep 17 18:14:15.578827 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:15.578785    2585 generic.go:356] "Generic (PLEG): container finished" podID="9a050689-5d4e-4949-a174-7180db32af12" containerID="2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5" exitCode=2
Sep 17 18:14:15.578970 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:15.578845    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerDied","Data":"2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5"}
Sep 17 18:14:18.136231 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:18.136189    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 18:14:18.140538 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:18.140503    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:14:20.088896 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.088873    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:14:20.176056 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.175983    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls\") pod \"9a050689-5d4e-4949-a174-7180db32af12\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") "
Sep 17 18:14:20.176190 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176054    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-kube-rbac-proxy-sar-config\") pod \"9a050689-5d4e-4949-a174-7180db32af12\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") "
Sep 17 18:14:20.176190 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176156    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-cabundle-cert\") pod \"9a050689-5d4e-4949-a174-7180db32af12\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") "
Sep 17 18:14:20.176262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176231    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gpt5\" (UniqueName: \"kubernetes.io/projected/9a050689-5d4e-4949-a174-7180db32af12-kube-api-access-7gpt5\") pod \"9a050689-5d4e-4949-a174-7180db32af12\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") "
Sep 17 18:14:20.176262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176255    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a050689-5d4e-4949-a174-7180db32af12-kserve-provision-location\") pod \"9a050689-5d4e-4949-a174-7180db32af12\" (UID: \"9a050689-5d4e-4949-a174-7180db32af12\") "
Sep 17 18:14:20.176427 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176402    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a050689-5d4e-4949-a174-7180db32af12" (UID: "9a050689-5d4e-4949-a174-7180db32af12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:20.176568 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176550    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.176635 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176610    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a050689-5d4e-4949-a174-7180db32af12" (UID: "9a050689-5d4e-4949-a174-7180db32af12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:20.176690 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.176650    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a050689-5d4e-4949-a174-7180db32af12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a050689-5d4e-4949-a174-7180db32af12" (UID: "9a050689-5d4e-4949-a174-7180db32af12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:20.178327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.178306    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a050689-5d4e-4949-a174-7180db32af12" (UID: "9a050689-5d4e-4949-a174-7180db32af12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:20.178425 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.178330    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a050689-5d4e-4949-a174-7180db32af12-kube-api-access-7gpt5" (OuterVolumeSpecName: "kube-api-access-7gpt5") pod "9a050689-5d4e-4949-a174-7180db32af12" (UID: "9a050689-5d4e-4949-a174-7180db32af12"). InnerVolumeSpecName "kube-api-access-7gpt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:20.277503 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.277472    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpt5\" (UniqueName: \"kubernetes.io/projected/9a050689-5d4e-4949-a174-7180db32af12-kube-api-access-7gpt5\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.277503 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.277498    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a050689-5d4e-4949-a174-7180db32af12-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.277503 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.277507    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a050689-5d4e-4949-a174-7180db32af12-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.277725 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.277516    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a050689-5d4e-4949-a174-7180db32af12-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.604288 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.604203    2585 generic.go:356] "Generic (PLEG): container finished" podID="9a050689-5d4e-4949-a174-7180db32af12" containerID="23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546" exitCode=0
Sep 17 18:14:20.604288 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.604263    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"
Sep 17 18:14:20.604483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.604284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerDied","Data":"23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546"}
Sep 17 18:14:20.604483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.604341    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw" event={"ID":"9a050689-5d4e-4949-a174-7180db32af12","Type":"ContainerDied","Data":"efaa1d98c45810b5f87e17306414f888cab0f33468741a955b0d52641fc66921"}
Sep 17 18:14:20.604483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.604371    2585 scope.go:117] "RemoveContainer" containerID="2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5"
Sep 17 18:14:20.613607 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.613590    2585 scope.go:117] "RemoveContainer" containerID="23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546"
Sep 17 18:14:20.621524 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.621506    2585 scope.go:117] "RemoveContainer" containerID="59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e"
Sep 17 18:14:20.625832 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.625791    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"]
Sep 17 18:14:20.631240 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.631216    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-x8jbw"]
Sep 17 18:14:20.635432 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.635413    2585 scope.go:117] "RemoveContainer" containerID="2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5"
Sep 17 18:14:20.635803 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:14:20.635778    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5\": container with ID starting with 2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5 not found: ID does not exist" containerID="2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5"
Sep 17 18:14:20.635915 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.635829    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5"} err="failed to get container status \"2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5\": rpc error: code = NotFound desc = could not find container \"2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5\": container with ID starting with 2591b7070e954e512266368018ed84d5c26ea8ad995613ff48451c6863a44cc5 not found: ID does not exist"
Sep 17 18:14:20.635915 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.635850    2585 scope.go:117] "RemoveContainer" containerID="23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546"
Sep 17 18:14:20.636144 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:14:20.636122    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546\": container with ID starting with 23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546 not found: ID does not exist" containerID="23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546"
Sep 17 18:14:20.636197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.636150    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546"} err="failed to get container status \"23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546\": rpc error: code = NotFound desc = could not find container \"23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546\": container with ID starting with 23745ccbb3f87ab0aac82e31e63ac11bbfc629ababdc0ffedbbe5f94b0fc4546 not found: ID does not exist"
Sep 17 18:14:20.636197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.636165    2585 scope.go:117] "RemoveContainer" containerID="59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e"
Sep 17 18:14:20.636437 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:14:20.636406    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e\": container with ID starting with 59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e not found: ID does not exist" containerID="59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e"
Sep 17 18:14:20.636499 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.636442    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e"} err="failed to get container status \"59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e\": rpc error: code = NotFound desc = could not find container \"59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e\": container with ID starting with 59a4bd4a438bf8ae13e2b645d459ddb857bd42a37c3bcacedff8e668c74b019e not found: ID does not exist"
Sep 17 18:14:20.960120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:20.960086    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a050689-5d4e-4949-a174-7180db32af12" path="/var/lib/kubelet/pods/9a050689-5d4e-4949-a174-7180db32af12/volumes"
Sep 17 18:14:26.897960 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:26.897927    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"]
Sep 17 18:14:26.898391 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:26.898167    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container" containerID="cri-o://77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3" gracePeriod=30
Sep 17 18:14:26.898391 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:26.898248    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" containerID="cri-o://0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87" gracePeriod=30
Sep 17 18:14:27.640937 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:27.640902    2585 generic.go:356] "Generic (PLEG): container finished" podID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerID="0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87" exitCode=2
Sep 17 18:14:27.641095 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:27.640976    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerDied","Data":"0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87"}
Sep 17 18:14:28.492219 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:28.492180    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:14:32.092098 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092059    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"]
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092452    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092464    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092479    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="storage-initializer"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092484    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="storage-initializer"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092491    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kube-rbac-proxy"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092496    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kube-rbac-proxy"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092511    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="storage-initializer"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092520    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="storage-initializer"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092531    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kube-rbac-proxy"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092538    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kube-rbac-proxy"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092545    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container"
Sep 17 18:14:32.092591 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092550    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container"
Sep 17 18:14:32.093278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092622    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kserve-container"
Sep 17 18:14:32.093278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092631    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kserve-container"
Sep 17 18:14:32.093278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092637    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7ae63e-b00a-458e-988f-7b7d50c7a27c" containerName="kube-rbac-proxy"
Sep 17 18:14:32.093278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.092643    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a050689-5d4e-4949-a174-7180db32af12" containerName="kube-rbac-proxy"
Sep 17 18:14:32.094971 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.094954    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.097698 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.097672    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 18:14:32.097836 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.097709    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:14:32.105890 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.105861    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"]
Sep 17 18:14:32.176764 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.176729    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.176964 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.176772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a259ae20-b5e1-4ebd-9da9-207bce304d30-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.176964 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.176833    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.176964 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.176860    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a259ae20-b5e1-4ebd-9da9-207bce304d30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.176964 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.176948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfkr\" (UniqueName: \"kubernetes.io/projected/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-api-access-rlfkr\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278013 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.277983    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a259ae20-b5e1-4ebd-9da9-207bce304d30-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278182 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278029    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278182 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278158    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a259ae20-b5e1-4ebd-9da9-207bce304d30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278302 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfkr\" (UniqueName: \"kubernetes.io/projected/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-api-access-rlfkr\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278353 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278316    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278431 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a259ae20-b5e1-4ebd-9da9-207bce304d30-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278721    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.278915 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.278893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.280965 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.280944    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a259ae20-b5e1-4ebd-9da9-207bce304d30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.286705 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.286685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfkr\" (UniqueName: \"kubernetes.io/projected/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-api-access-rlfkr\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.406930 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.406901    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:32.677067 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:32.677043    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"]
Sep 17 18:14:32.678770 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:14:32.678740    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda259ae20_b5e1_4ebd_9da9_207bce304d30.slice/crio-742014eb6d4eff6aa9d39f597599cd82fd0cab842a959e4b34bfdb4c45067493 WatchSource:0}: Error finding container 742014eb6d4eff6aa9d39f597599cd82fd0cab842a959e4b34bfdb4c45067493: Status 404 returned error can't find the container with id 742014eb6d4eff6aa9d39f597599cd82fd0cab842a959e4b34bfdb4c45067493
Sep 17 18:14:33.492643 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:33.492606    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:14:33.670989 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:33.670958    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerStarted","Data":"0851333ac8a99b9c0901d199799667630ea72f9c321ddfc1e80c6df723350907"}
Sep 17 18:14:33.671157 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:33.671014    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerStarted","Data":"742014eb6d4eff6aa9d39f597599cd82fd0cab842a959e4b34bfdb4c45067493"}
Sep 17 18:14:37.692447 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:37.692412    2585 generic.go:356] "Generic (PLEG): container finished" podID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerID="0851333ac8a99b9c0901d199799667630ea72f9c321ddfc1e80c6df723350907" exitCode=0
Sep 17 18:14:37.692831 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:37.692484    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerDied","Data":"0851333ac8a99b9c0901d199799667630ea72f9c321ddfc1e80c6df723350907"}
Sep 17 18:14:38.492982 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.492936    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:14:38.493185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.493086    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:14:38.700270 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.700238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerStarted","Data":"12b5b9f0e6de8be5c6ce89955c766571929af34798c6db27cfc39a4a58ddb830"}
Sep 17 18:14:38.700674 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.700299    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerStarted","Data":"7dd22635c71da3e9f410b7cd01d56ef433108a6e5a98183d3c75199fe6152176"}
Sep 17 18:14:38.700674 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.700573    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:38.700785 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.700724    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:38.701908 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.701882    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:14:38.719575 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:38.719531    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podStartSLOduration=2.319808246 podStartE2EDuration="6.719518454s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.399710208s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:38.718658592 +0000 UTC m=+2011.909996280" watchObservedRunningTime="2026-09-17 18:14:38.719518454 +0000 UTC m=+2011.910856139"
Sep 17 18:14:39.705257 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:39.705223    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:14:43.492452 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:43.492407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:14:44.709934 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:44.709907    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:44.710427 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:44.710390    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:14:48.492791 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:48.492752    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:14:53.492242 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:53.492199    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:14:54.710977 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:54.710949    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:14:57.549625 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.549602    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:14:57.682377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682304    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trfrq\" (UniqueName: \"kubernetes.io/projected/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-api-access-trfrq\") pod \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") "
Sep 17 18:14:57.682377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682362    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls\") pod \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") "
Sep 17 18:14:57.682606 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682385    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-cabundle-cert\") pod \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") "
Sep 17 18:14:57.682606 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682438    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kserve-provision-location\") pod \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") "
Sep 17 18:14:57.682606 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682483    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-rbac-proxy-sar-config\") pod \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\" (UID: \"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5\") "
Sep 17 18:14:57.682873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682847    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" (UID: "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:57.682873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.682856    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" (UID: "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:57.684654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.684632    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" (UID: "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:57.684733 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.684716    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-api-access-trfrq" (OuterVolumeSpecName: "kube-api-access-trfrq") pod "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" (UID: "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5"). InnerVolumeSpecName "kube-api-access-trfrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:57.692546 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.692520    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" (UID: "2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:57.783511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.783488    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:57.783511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.783509    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfrq\" (UniqueName: \"kubernetes.io/projected/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kube-api-access-trfrq\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:57.783632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.783519    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:57.783632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.783529    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:57.783632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.783537    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:14:57.791359 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.791332    2585 generic.go:356] "Generic (PLEG): container finished" podID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerID="77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3" exitCode=137
Sep 17 18:14:57.791451 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.791396    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"
Sep 17 18:14:57.791451 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.791412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerDied","Data":"77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3"}
Sep 17 18:14:57.791527 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.791473    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2" event={"ID":"2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5","Type":"ContainerDied","Data":"f8b252ac505d90a243f6b8708f2d1e603f4520db31b0fe512713b6be3876d539"}
Sep 17 18:14:57.791527 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.791503    2585 scope.go:117] "RemoveContainer" containerID="0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87"
Sep 17 18:14:57.802158 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.801916    2585 scope.go:117] "RemoveContainer" containerID="77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3"
Sep 17 18:14:57.810246 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.810228    2585 scope.go:117] "RemoveContainer" containerID="80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b"
Sep 17 18:14:57.818080 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818064    2585 scope.go:117] "RemoveContainer" containerID="0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87"
Sep 17 18:14:57.818133 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818096    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"]
Sep 17 18:14:57.818376 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:14:57.818359    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87\": container with ID starting with 0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87 not found: ID does not exist" containerID="0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87"
Sep 17 18:14:57.818449 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818382    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87"} err="failed to get container status \"0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87\": rpc error: code = NotFound desc = could not find container \"0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87\": container with ID starting with 0a8c707678df61fb33a04bc12da985bd0b4e2cad7730d80af2d08cfd12de4a87 not found: ID does not exist"
Sep 17 18:14:57.818449 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818398    2585 scope.go:117] "RemoveContainer" containerID="77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3"
Sep 17 18:14:57.818657 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:14:57.818642    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3\": container with ID starting with 77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3 not found: ID does not exist" containerID="77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3"
Sep 17 18:14:57.818699 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818662    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3"} err="failed to get container status \"77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3\": rpc error: code = NotFound desc = could not find container \"77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3\": container with ID starting with 77e33a4de6de5ea3ee8d851e9d0a8c1123f1f306679d22f0f46922e7e83c3af3 not found: ID does not exist"
Sep 17 18:14:57.818699 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818677    2585 scope.go:117] "RemoveContainer" containerID="80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b"
Sep 17 18:14:57.818940 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:14:57.818921    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b\": container with ID starting with 80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b not found: ID does not exist" containerID="80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b"
Sep 17 18:14:57.818998 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.818947    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b"} err="failed to get container status \"80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b\": rpc error: code = NotFound desc = could not find container \"80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b\": container with ID starting with 80769f8a010c304b6e98626ba32cd8c4554a3ed6399639abaa62932ead08230b not found: ID does not exist"
Sep 17 18:14:57.832322 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:57.832298    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-x68r2"]
Sep 17 18:14:58.960653 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:14:58.960622    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" path="/var/lib/kubelet/pods/2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5/volumes"
Sep 17 18:15:11.600124 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:11.600088    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"]
Sep 17 18:15:11.600559 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:11.600376    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container" containerID="cri-o://7dd22635c71da3e9f410b7cd01d56ef433108a6e5a98183d3c75199fe6152176" gracePeriod=30
Sep 17 18:15:11.600559 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:11.600457    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" containerID="cri-o://12b5b9f0e6de8be5c6ce89955c766571929af34798c6db27cfc39a4a58ddb830" gracePeriod=30
Sep 17 18:15:11.857906 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:11.857827    2585 generic.go:356] "Generic (PLEG): container finished" podID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerID="12b5b9f0e6de8be5c6ce89955c766571929af34798c6db27cfc39a4a58ddb830" exitCode=2
Sep 17 18:15:11.857906 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:11.857836    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerDied","Data":"12b5b9f0e6de8be5c6ce89955c766571929af34798c6db27cfc39a4a58ddb830"}
Sep 17 18:15:14.705400 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:14.705364    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:15:16.811879 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.811840    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"]
Sep 17 18:15:16.812317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812219    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container"
Sep 17 18:15:16.812317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812231    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container"
Sep 17 18:15:16.812317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812242    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy"
Sep 17 18:15:16.812317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812248    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy"
Sep 17 18:15:16.812317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812264    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="storage-initializer"
Sep 17 18:15:16.812317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812269    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="storage-initializer"
Sep 17 18:15:16.812507 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812335    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kserve-container"
Sep 17 18:15:16.812507 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.812344    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb57aa1-ad4c-40ac-b8b1-9330d4492fa5" containerName="kube-rbac-proxy"
Sep 17 18:15:16.815712 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.815693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:16.818364 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.818340    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 18:15:16.818483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.818407    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 18:15:16.825788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.825762    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"]
Sep 17 18:15:16.934659 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.934629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2484\" (UniqueName: \"kubernetes.io/projected/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-api-access-d2484\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:16.934853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.934681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:16.934853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.934744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:16.934853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.934838    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db79adc0-0ad9-40a7-9f3c-3b48621a9794-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:16.934995 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:16.934861    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.036219 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.036187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db79adc0-0ad9-40a7-9f3c-3b48621a9794-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.036219 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.036221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.036429 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.036256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2484\" (UniqueName: \"kubernetes.io/projected/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-api-access-d2484\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.036429 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.036318    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.036429 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.036347    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.036764 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.036743    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.037129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.037083    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.037129 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.037083    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.038861 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.038840    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db79adc0-0ad9-40a7-9f3c-3b48621a9794-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.049407 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.049387    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2484\" (UniqueName: \"kubernetes.io/projected/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-api-access-d2484\") pod \"isvc-triton-predictor-5d7db58676-6vnkh\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.127394 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.127369    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:15:17.399203 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.399177    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"]
Sep 17 18:15:17.400755 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:15:17.400729    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb79adc0_0ad9_40a7_9f3c_3b48621a9794.slice/crio-459209b476ff798bd4782fa205d2e383b7f8a04e2217521ee5530fff4ee24601 WatchSource:0}: Error finding container 459209b476ff798bd4782fa205d2e383b7f8a04e2217521ee5530fff4ee24601: Status 404 returned error can't find the container with id 459209b476ff798bd4782fa205d2e383b7f8a04e2217521ee5530fff4ee24601
Sep 17 18:15:17.888368 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.888335    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerStarted","Data":"d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e"}
Sep 17 18:15:17.888736 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:17.888397    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerStarted","Data":"459209b476ff798bd4782fa205d2e383b7f8a04e2217521ee5530fff4ee24601"}
Sep 17 18:15:19.705720 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:19.705646    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:15:21.908898 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:21.908803    2585 generic.go:356] "Generic (PLEG): container finished" podID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerID="d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e" exitCode=0
Sep 17 18:15:21.908898 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:21.908841    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerDied","Data":"d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e"}
Sep 17 18:15:24.706940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:24.706204    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:15:24.706940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:24.706362    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:15:29.706335 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:29.705895    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:15:34.706111 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:34.705938    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:15:39.706261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:39.706213    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:15:42.072430 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.072371    2585 generic.go:356] "Generic (PLEG): container finished" podID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerID="7dd22635c71da3e9f410b7cd01d56ef433108a6e5a98183d3c75199fe6152176" exitCode=137
Sep 17 18:15:42.072891 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.072435    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerDied","Data":"7dd22635c71da3e9f410b7cd01d56ef433108a6e5a98183d3c75199fe6152176"}
Sep 17 18:15:42.328960 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.328900    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:15:42.493735 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.493700    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a259ae20-b5e1-4ebd-9da9-207bce304d30-kserve-provision-location\") pod \"a259ae20-b5e1-4ebd-9da9-207bce304d30\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") "
Sep 17 18:15:42.493949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.493751    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a259ae20-b5e1-4ebd-9da9-207bce304d30-proxy-tls\") pod \"a259ae20-b5e1-4ebd-9da9-207bce304d30\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") "
Sep 17 18:15:42.493949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.493782    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlfkr\" (UniqueName: \"kubernetes.io/projected/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-api-access-rlfkr\") pod \"a259ae20-b5e1-4ebd-9da9-207bce304d30\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") "
Sep 17 18:15:42.493949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.493852    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-cabundle-cert\") pod \"a259ae20-b5e1-4ebd-9da9-207bce304d30\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") "
Sep 17 18:15:42.493949 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.493925    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-rbac-proxy-sar-config\") pod \"a259ae20-b5e1-4ebd-9da9-207bce304d30\" (UID: \"a259ae20-b5e1-4ebd-9da9-207bce304d30\") "
Sep 17 18:15:42.494290 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.494263    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a259ae20-b5e1-4ebd-9da9-207bce304d30" (UID: "a259ae20-b5e1-4ebd-9da9-207bce304d30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:42.494363 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.494312    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a259ae20-b5e1-4ebd-9da9-207bce304d30" (UID: "a259ae20-b5e1-4ebd-9da9-207bce304d30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:42.496830 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.496787    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a259ae20-b5e1-4ebd-9da9-207bce304d30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a259ae20-b5e1-4ebd-9da9-207bce304d30" (UID: "a259ae20-b5e1-4ebd-9da9-207bce304d30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:42.497751 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.497716    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-api-access-rlfkr" (OuterVolumeSpecName: "kube-api-access-rlfkr") pod "a259ae20-b5e1-4ebd-9da9-207bce304d30" (UID: "a259ae20-b5e1-4ebd-9da9-207bce304d30"). InnerVolumeSpecName "kube-api-access-rlfkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:42.505261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.504591    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a259ae20-b5e1-4ebd-9da9-207bce304d30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a259ae20-b5e1-4ebd-9da9-207bce304d30" (UID: "a259ae20-b5e1-4ebd-9da9-207bce304d30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:42.595649 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.595555    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:42.595649 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.595584    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a259ae20-b5e1-4ebd-9da9-207bce304d30-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:42.595649 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.595599    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a259ae20-b5e1-4ebd-9da9-207bce304d30-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:42.595649 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.595611    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfkr\" (UniqueName: \"kubernetes.io/projected/a259ae20-b5e1-4ebd-9da9-207bce304d30-kube-api-access-rlfkr\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:42.595649 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:42.595624    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a259ae20-b5e1-4ebd-9da9-207bce304d30-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:43.081941 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.081894    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5" event={"ID":"a259ae20-b5e1-4ebd-9da9-207bce304d30","Type":"ContainerDied","Data":"742014eb6d4eff6aa9d39f597599cd82fd0cab842a959e4b34bfdb4c45067493"}
Sep 17 18:15:43.082376 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.081988    2585 scope.go:117] "RemoveContainer" containerID="12b5b9f0e6de8be5c6ce89955c766571929af34798c6db27cfc39a4a58ddb830"
Sep 17 18:15:43.082376 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.082247    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"
Sep 17 18:15:43.094084 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.094052    2585 scope.go:117] "RemoveContainer" containerID="7dd22635c71da3e9f410b7cd01d56ef433108a6e5a98183d3c75199fe6152176"
Sep 17 18:15:43.104357 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.104327    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"]
Sep 17 18:15:43.107007 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.106969    2585 scope.go:117] "RemoveContainer" containerID="0851333ac8a99b9c0901d199799667630ea72f9c321ddfc1e80c6df723350907"
Sep 17 18:15:43.111537 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:43.111482    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c5hr5"]
Sep 17 18:15:44.964488 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:44.964441    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" path="/var/lib/kubelet/pods/a259ae20-b5e1-4ebd-9da9-207bce304d30/volumes"
Sep 17 18:15:45.623137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:45.622324    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"]
Sep 17 18:15:45.623137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:45.622632    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" containerID="cri-o://cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0" gracePeriod=30
Sep 17 18:15:45.623137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:45.622838    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kube-rbac-proxy" containerID="cri-o://94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166" gracePeriod=30
Sep 17 18:15:46.109884 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:46.109784    2585 generic.go:356] "Generic (PLEG): container finished" podID="43488c7d-449f-49c0-a744-3a5c139af29d" containerID="94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166" exitCode=2
Sep 17 18:15:46.110301 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:46.109902    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerDied","Data":"94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166"}
Sep 17 18:15:49.140976 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:49.140929    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 18:15:49.145500 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:49.145459    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.923604    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"]
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924243    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="storage-initializer"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924263    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="storage-initializer"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924279    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924288    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924317    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924326    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924420    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kserve-container"
Sep 17 18:15:50.924983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.924431    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a259ae20-b5e1-4ebd-9da9-207bce304d30" containerName="kube-rbac-proxy"
Sep 17 18:15:50.947778 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.947750    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"]
Sep 17 18:15:50.947941 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.947921    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:50.952588 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.952024    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:15:50.952588 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.952073    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:15:50.979459 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.978109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlfn\" (UniqueName: \"kubernetes.io/projected/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-api-access-kmlfn\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:50.979459 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.978193    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:50.979459 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.978229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:50.979459 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.978281    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3dd24f6-d921-40ae-98c2-4116c1533885-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:50.979459 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:50.978313    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.079171 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.079057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3dd24f6-d921-40ae-98c2-4116c1533885-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.079171 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.079111    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.079417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.079181    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlfn\" (UniqueName: \"kubernetes.io/projected/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-api-access-kmlfn\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.079417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.079240    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.079417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.079272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.079587 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.079549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3dd24f6-d921-40ae-98c2-4116c1533885-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.080140 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:15:51.079994    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:15:51.080140 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.080053    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.080140 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:15:51.080062    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls podName:f3dd24f6-d921-40ae-98c2-4116c1533885 nodeName:}" failed. No retries permitted until 2026-09-17 18:15:51.580044069 +0000 UTC m=+2084.771381731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls") pod "isvc-canary-rollback-predictor-76cbc5464-j84g8" (UID: "f3dd24f6-d921-40ae-98c2-4116c1533885") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:15:51.080410 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.080342    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.089510 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.089459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlfn\" (UniqueName: \"kubernetes.io/projected/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-api-access-kmlfn\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.584352 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.584260    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.591698 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.589876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-j84g8\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:51.641352 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.641278    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:15:51.685097 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.685067    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls\") pod \"43488c7d-449f-49c0-a744-3a5c139af29d\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") "
Sep 17 18:15:51.685261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.685109    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-cabundle-cert\") pod \"43488c7d-449f-49c0-a744-3a5c139af29d\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") "
Sep 17 18:15:51.685261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.685137    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-547hr\" (UniqueName: \"kubernetes.io/projected/43488c7d-449f-49c0-a744-3a5c139af29d-kube-api-access-547hr\") pod \"43488c7d-449f-49c0-a744-3a5c139af29d\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") "
Sep 17 18:15:51.685537 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.685511    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43488c7d-449f-49c0-a744-3a5c139af29d" (UID: "43488c7d-449f-49c0-a744-3a5c139af29d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:51.685631 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.685586    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-kube-rbac-proxy-sar-config\") pod \"43488c7d-449f-49c0-a744-3a5c139af29d\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") "
Sep 17 18:15:51.685696 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.685646    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43488c7d-449f-49c0-a744-3a5c139af29d-kserve-provision-location\") pod \"43488c7d-449f-49c0-a744-3a5c139af29d\" (UID: \"43488c7d-449f-49c0-a744-3a5c139af29d\") "
Sep 17 18:15:51.686090 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.686063    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43488c7d-449f-49c0-a744-3a5c139af29d" (UID: "43488c7d-449f-49c0-a744-3a5c139af29d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:51.686196 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.686155    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43488c7d-449f-49c0-a744-3a5c139af29d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43488c7d-449f-49c0-a744-3a5c139af29d" (UID: "43488c7d-449f-49c0-a744-3a5c139af29d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:51.686848 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.686604    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:51.686848 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.686627    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43488c7d-449f-49c0-a744-3a5c139af29d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:51.686848 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.686638    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43488c7d-449f-49c0-a744-3a5c139af29d-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:51.688256 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.688197    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43488c7d-449f-49c0-a744-3a5c139af29d" (UID: "43488c7d-449f-49c0-a744-3a5c139af29d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:51.688618 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.688592    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43488c7d-449f-49c0-a744-3a5c139af29d-kube-api-access-547hr" (OuterVolumeSpecName: "kube-api-access-547hr") pod "43488c7d-449f-49c0-a744-3a5c139af29d" (UID: "43488c7d-449f-49c0-a744-3a5c139af29d"). InnerVolumeSpecName "kube-api-access-547hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:51.788182 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.788079    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43488c7d-449f-49c0-a744-3a5c139af29d-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:51.788182 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.788114    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-547hr\" (UniqueName: \"kubernetes.io/projected/43488c7d-449f-49c0-a744-3a5c139af29d-kube-api-access-547hr\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:15:51.866523 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:51.866483    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:52.154084 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.154051    2585 generic.go:356] "Generic (PLEG): container finished" podID="43488c7d-449f-49c0-a744-3a5c139af29d" containerID="cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0" exitCode=0
Sep 17 18:15:52.154631 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.154126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerDied","Data":"cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0"}
Sep 17 18:15:52.154832 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.154792    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv" event={"ID":"43488c7d-449f-49c0-a744-3a5c139af29d","Type":"ContainerDied","Data":"55342bc8dd034b672bce550b66a50fccdd9519f92520d14032b27bd76bb02916"}
Sep 17 18:15:52.154923 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.154905    2585 scope.go:117] "RemoveContainer" containerID="94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166"
Sep 17 18:15:52.155019 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.154222    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"
Sep 17 18:15:52.167714 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.167665    2585 scope.go:117] "RemoveContainer" containerID="cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0"
Sep 17 18:15:52.175355 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.175335    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"]
Sep 17 18:15:52.182653 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.182627    2585 scope.go:117] "RemoveContainer" containerID="dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053"
Sep 17 18:15:52.184274 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.184252    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"]
Sep 17 18:15:52.189226 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.189191    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x45wv"]
Sep 17 18:15:52.196439 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.196423    2585 scope.go:117] "RemoveContainer" containerID="94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166"
Sep 17 18:15:52.196736 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:15:52.196699    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166\": container with ID starting with 94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166 not found: ID does not exist" containerID="94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166"
Sep 17 18:15:52.196851 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.196748    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166"} err="failed to get container status \"94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166\": rpc error: code = NotFound desc = could not find container \"94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166\": container with ID starting with 94c9f09085b82523b8d29a101941426c22d991d0025817360a89e643b892a166 not found: ID does not exist"
Sep 17 18:15:52.196851 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.196771    2585 scope.go:117] "RemoveContainer" containerID="cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0"
Sep 17 18:15:52.197129 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:15:52.197102    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0\": container with ID starting with cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0 not found: ID does not exist" containerID="cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0"
Sep 17 18:15:52.197216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.197141    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0"} err="failed to get container status \"cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0\": rpc error: code = NotFound desc = could not find container \"cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0\": container with ID starting with cb8de60cc7cb03c9eb9ddd04e5150b673a7938d5254f23e2ceaef814838439c0 not found: ID does not exist"
Sep 17 18:15:52.197216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.197164    2585 scope.go:117] "RemoveContainer" containerID="dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053"
Sep 17 18:15:52.197509 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:15:52.197484    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053\": container with ID starting with dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053 not found: ID does not exist" containerID="dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053"
Sep 17 18:15:52.197595 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.197518    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053"} err="failed to get container status \"dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053\": rpc error: code = NotFound desc = could not find container \"dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053\": container with ID starting with dde6a5e1ef07817af38de53a3f8a376723bd5b05acc1ba467837e81ae7851053 not found: ID does not exist"
Sep 17 18:15:52.962830 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:52.962763    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" path="/var/lib/kubelet/pods/43488c7d-449f-49c0-a744-3a5c139af29d/volumes"
Sep 17 18:15:53.165041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:53.164931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerStarted","Data":"deff4fbb6a054e1fba1031707358c2002dc73a1f9f9c5029c48a2368f92510ac"}
Sep 17 18:15:53.165041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:53.164997    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerStarted","Data":"d5b5ab08332024e39a4ebbb5c70e1d385d078c2c7f2cbc116b0cb8688b9fd281"}
Sep 17 18:15:57.194700 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:57.194652    2585 generic.go:356] "Generic (PLEG): container finished" podID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerID="deff4fbb6a054e1fba1031707358c2002dc73a1f9f9c5029c48a2368f92510ac" exitCode=0
Sep 17 18:15:57.195236 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:57.194723    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerDied","Data":"deff4fbb6a054e1fba1031707358c2002dc73a1f9f9c5029c48a2368f92510ac"}
Sep 17 18:15:58.206205 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:58.205773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerStarted","Data":"c6a4551b7b9cd432943c45e29491027ac62429c75e3c7f3cf567b0a8d84f3422"}
Sep 17 18:15:58.206205 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:58.205872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerStarted","Data":"768be3ce3c4910f66738ac515b5114523cde99c11af88ccbb2ad54c520e46489"}
Sep 17 18:15:58.206205 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:58.206095    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:58.206205 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:58.206150    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:15:58.207713 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:58.207674    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:15:58.228497 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:58.228451    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podStartSLOduration=4.358956548 podStartE2EDuration="8.22843291s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869476362s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:58.22653718 +0000 UTC m=+2091.417874867" watchObservedRunningTime="2026-09-17 18:15:58.22843291 +0000 UTC m=+2091.419770596"
Sep 17 18:15:59.213898 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:15:59.213468    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:16:04.219215 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:04.219169    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:16:04.220087 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:04.220046    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:16:14.221051 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:14.221009    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:16:24.221137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:24.221091    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:16:34.220833 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:34.220769    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:16:44.220397 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:44.220338    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:16:54.221147 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:16:54.221039    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:17:04.221037 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:04.220999    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:17:14.220569 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:14.220532    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:17:15.640131 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:15.640093    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"]
Sep 17 18:17:15.640612 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:15.640384    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" containerID="cri-o://768be3ce3c4910f66738ac515b5114523cde99c11af88ccbb2ad54c520e46489" gracePeriod=30
Sep 17 18:17:15.640612 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:15.640433    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" containerID="cri-o://c6a4551b7b9cd432943c45e29491027ac62429c75e3c7f3cf567b0a8d84f3422" gracePeriod=30
Sep 17 18:17:15.640741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:15.640676    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:17:17.539845 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:17.539792    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:17:17.540252 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:17.540127    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:17:17.674025 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:17.673995    2585 generic.go:356] "Generic (PLEG): container finished" podID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerID="c6a4551b7b9cd432943c45e29491027ac62429c75e3c7f3cf567b0a8d84f3422" exitCode=2
Sep 17 18:17:17.674181 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:17.674066    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerDied","Data":"c6a4551b7b9cd432943c45e29491027ac62429c75e3c7f3cf567b0a8d84f3422"}
Sep 17 18:17:18.680562 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:18.680524    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerStarted","Data":"48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e"}
Sep 17 18:17:18.680970 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:18.680627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerStarted","Data":"2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73"}
Sep 17 18:17:18.680970 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:18.680704    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:17:18.701318 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:18.701266    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" podStartSLOduration=2.095264366 podStartE2EDuration="2m2.701247663s" totalImagesPullingTime="1m56.439101906s" totalInitContainerRuntime="4.166881391s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:18.69852225 +0000 UTC m=+2171.889859936" watchObservedRunningTime="2026-09-17 18:17:18.701247663 +0000 UTC m=+2171.892585349"
Sep 17 18:17:19.213992 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:19.213944    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:17:19.685462 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:19.685428    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:17:19.686709 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:19.686679    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:17:20.690453 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.690413    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:17:20.920913 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.920879    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"]
Sep 17 18:17:20.921279 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921266    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container"
Sep 17 18:17:20.921337 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921281    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container"
Sep 17 18:17:20.921337 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921298    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kube-rbac-proxy"
Sep 17 18:17:20.921337 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921307    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kube-rbac-proxy"
Sep 17 18:17:20.921337 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921322    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="storage-initializer"
Sep 17 18:17:20.921337 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921330    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="storage-initializer"
Sep 17 18:17:20.921512 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921432    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kube-rbac-proxy"
Sep 17 18:17:20.921512 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.921444    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="43488c7d-449f-49c0-a744-3a5c139af29d" containerName="kserve-container"
Sep 17 18:17:20.938728 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.938695    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"]
Sep 17 18:17:20.938880 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:20.938830    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.024948 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.024916    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bbz\" (UniqueName: \"kubernetes.io/projected/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-api-access-r4bbz\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.025109 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.024954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.025109 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.025071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.025184 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.025106    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.025184 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.025158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.125730 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.125694    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.125925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.125737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.125925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.125773    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.125925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.125802    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bbz\" (UniqueName: \"kubernetes.io/projected/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-api-access-r4bbz\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.125925 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.125868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.126353 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.126331    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.126460 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.126413    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.126569 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.126547    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.128531 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.128513    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.133869 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.133845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bbz\" (UniqueName: \"kubernetes.io/projected/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-api-access-r4bbz\") pod \"isvc-canary-rollback-predictor-76cbc5464-5bh2w\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.249712 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.249628    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:21.577005 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.576971    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"]
Sep 17 18:17:21.578834 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:17:21.578786    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e7ce65_a8b1_4c8c_862e_17b5c9f24c4f.slice/crio-b09599d2535a9ee60797d483669f3870ba0be2ee4044cb3226a7dc1e66e2b399 WatchSource:0}: Error finding container b09599d2535a9ee60797d483669f3870ba0be2ee4044cb3226a7dc1e66e2b399: Status 404 returned error can't find the container with id b09599d2535a9ee60797d483669f3870ba0be2ee4044cb3226a7dc1e66e2b399
Sep 17 18:17:21.697750 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.697721    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerStarted","Data":"fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee"}
Sep 17 18:17:21.698187 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:21.697772    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerStarted","Data":"b09599d2535a9ee60797d483669f3870ba0be2ee4044cb3226a7dc1e66e2b399"}
Sep 17 18:17:24.213650 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:24.213608    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:17:25.641891 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:25.641857    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:17:25.694955 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:25.694928    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:17:25.695606 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:25.695585    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:17:26.724454 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:26.724421    2585 generic.go:356] "Generic (PLEG): container finished" podID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerID="fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee" exitCode=0
Sep 17 18:17:26.724870 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:26.724490    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerDied","Data":"fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee"}
Sep 17 18:17:27.731159 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:27.731122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerStarted","Data":"aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f"}
Sep 17 18:17:27.731555 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:27.731192    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerStarted","Data":"932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d"}
Sep 17 18:17:27.731555 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:27.731385    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:27.749630 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:27.749587    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podStartSLOduration=3.519575219 podStartE2EDuration="7.749573833s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.229998614s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:27.747942418 +0000 UTC m=+2180.939280103" watchObservedRunningTime="2026-09-17 18:17:27.749573833 +0000 UTC m=+2180.940911518"
Sep 17 18:17:28.736975 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:28.736942    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:28.738305 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:28.738277    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:17:29.210832 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.210787    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"]
Sep 17 18:17:29.211034 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.211010    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kserve-container" containerID="cri-o://2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73" gracePeriod=30
Sep 17 18:17:29.211128 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.211073    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kube-rbac-proxy" containerID="cri-o://48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e" gracePeriod=30
Sep 17 18:17:29.213770 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.213736    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:17:29.213907 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.213865    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:17:29.743128 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.743095    2585 generic.go:356] "Generic (PLEG): container finished" podID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerID="48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e" exitCode=2
Sep 17 18:17:29.743546 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.743170    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerDied","Data":"48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e"}
Sep 17 18:17:29.743546 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:29.743494    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:17:30.691581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:30.691541    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 18:17:32.194729 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.194704    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:17:32.334837 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.334741    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-rbac-proxy-sar-config\") pod \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") "
Sep 17 18:17:32.334837 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.334801    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kserve-provision-location\") pod \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") "
Sep 17 18:17:32.335058 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.334850    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-cabundle-cert\") pod \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") "
Sep 17 18:17:32.335058 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.334886    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2484\" (UniqueName: \"kubernetes.io/projected/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-api-access-d2484\") pod \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") "
Sep 17 18:17:32.335058 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.334916    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db79adc0-0ad9-40a7-9f3c-3b48621a9794-proxy-tls\") pod \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\" (UID: \"db79adc0-0ad9-40a7-9f3c-3b48621a9794\") "
Sep 17 18:17:32.335231 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.335195    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db79adc0-0ad9-40a7-9f3c-3b48621a9794" (UID: "db79adc0-0ad9-40a7-9f3c-3b48621a9794"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:32.335231 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.335216    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db79adc0-0ad9-40a7-9f3c-3b48621a9794" (UID: "db79adc0-0ad9-40a7-9f3c-3b48621a9794"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:32.335310 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.335217    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db79adc0-0ad9-40a7-9f3c-3b48621a9794" (UID: "db79adc0-0ad9-40a7-9f3c-3b48621a9794"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:32.337099 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.337072    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-api-access-d2484" (OuterVolumeSpecName: "kube-api-access-d2484") pod "db79adc0-0ad9-40a7-9f3c-3b48621a9794" (UID: "db79adc0-0ad9-40a7-9f3c-3b48621a9794"). InnerVolumeSpecName "kube-api-access-d2484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:32.337099 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.337072    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db79adc0-0ad9-40a7-9f3c-3b48621a9794-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db79adc0-0ad9-40a7-9f3c-3b48621a9794" (UID: "db79adc0-0ad9-40a7-9f3c-3b48621a9794"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:32.436015 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.435990    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2484\" (UniqueName: \"kubernetes.io/projected/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-api-access-d2484\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:32.436015 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.436013    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db79adc0-0ad9-40a7-9f3c-3b48621a9794-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:32.436015 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.436022    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:32.436264 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.436031    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db79adc0-0ad9-40a7-9f3c-3b48621a9794-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:32.436264 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.436040    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db79adc0-0ad9-40a7-9f3c-3b48621a9794-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:32.760202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.760101    2585 generic.go:356] "Generic (PLEG): container finished" podID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerID="2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73" exitCode=0
Sep 17 18:17:32.760202 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.760186    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerDied","Data":"2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73"}
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.760213    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.760234    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh" event={"ID":"db79adc0-0ad9-40a7-9f3c-3b48621a9794","Type":"ContainerDied","Data":"459209b476ff798bd4782fa205d2e383b7f8a04e2217521ee5530fff4ee24601"}
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.760261    2585 scope.go:117] "RemoveContainer" containerID="48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.769250    2585 scope.go:117] "RemoveContainer" containerID="2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.777233    2585 scope.go:117] "RemoveContainer" containerID="d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.781117    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"]
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.784073    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6vnkh"]
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.785490    2585 scope.go:117] "RemoveContainer" containerID="48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:17:32.785747    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e\": container with ID starting with 48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e not found: ID does not exist" containerID="48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.785772    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e"} err="failed to get container status \"48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e\": rpc error: code = NotFound desc = could not find container \"48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e\": container with ID starting with 48c13fda6c337dc5b294de9e628bf1b56b3f74a98074358727d73724ffa1158e not found: ID does not exist"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.785788    2585 scope.go:117] "RemoveContainer" containerID="2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:17:32.786046    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73\": container with ID starting with 2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73 not found: ID does not exist" containerID="2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.786063    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73"} err="failed to get container status \"2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73\": rpc error: code = NotFound desc = could not find container \"2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73\": container with ID starting with 2559eefac8fc87557ebd89dcd19c17fdaa82b4ab8321b4b993be355fc15cff73 not found: ID does not exist"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.786075    2585 scope.go:117] "RemoveContainer" containerID="d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:17:32.786271    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e\": container with ID starting with d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e not found: ID does not exist" containerID="d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e"
Sep 17 18:17:32.916267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.786285    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e"} err="failed to get container status \"d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e\": rpc error: code = NotFound desc = could not find container \"d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e\": container with ID starting with d72562dad3d4666952c7dfad80465323593cfed9873f92f00bb47f360f39eb1e not found: ID does not exist"
Sep 17 18:17:32.959919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:32.959896    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" path="/var/lib/kubelet/pods/db79adc0-0ad9-40a7-9f3c-3b48621a9794/volumes"
Sep 17 18:17:34.213798 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.213754    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:17:34.393161 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393129    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"]
Sep 17 18:17:34.393516 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393504    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kube-rbac-proxy"
Sep 17 18:17:34.393561 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393519    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kube-rbac-proxy"
Sep 17 18:17:34.393561 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393533    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kserve-container"
Sep 17 18:17:34.393561 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393538    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kserve-container"
Sep 17 18:17:34.393561 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393551    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="storage-initializer"
Sep 17 18:17:34.393561 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393557    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="storage-initializer"
Sep 17 18:17:34.393724 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393614    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kube-rbac-proxy"
Sep 17 18:17:34.393724 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.393624    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="db79adc0-0ad9-40a7-9f3c-3b48621a9794" containerName="kserve-container"
Sep 17 18:17:34.429092 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.429057    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"]
Sep 17 18:17:34.429262 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.429184    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.431786 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.431765    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 18:17:34.431946 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.431765    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 18:17:34.556747 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.556663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.556747 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.556703    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2665f22-12be-44a7-a8fd-678d918e6db3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.556975 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.556770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4psj\" (UniqueName: \"kubernetes.io/projected/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-api-access-d4psj\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.556975 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.556884    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.556975 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.556929    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.657721 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.657685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.657919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.657730    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.657919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.657762    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.657919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.657779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2665f22-12be-44a7-a8fd-678d918e6db3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.657919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.657844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4psj\" (UniqueName: \"kubernetes.io/projected/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-api-access-d4psj\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.657919 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:17:34.657848    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 18:17:34.657919 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:17:34.657911    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls podName:f2665f22-12be-44a7-a8fd-678d918e6db3 nodeName:}" failed. No retries permitted until 2026-09-17 18:17:35.157892104 +0000 UTC m=+2188.349229768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-p4x9l" (UID: "f2665f22-12be-44a7-a8fd-678d918e6db3") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 18:17:34.658244 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.658211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2665f22-12be-44a7-a8fd-678d918e6db3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.658545 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.658523    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.658598 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.658523    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.666572 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.666553    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4psj\" (UniqueName: \"kubernetes.io/projected/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-api-access-d4psj\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:34.747847 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.747797    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:17:34.748381 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:34.748357    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:17:35.162348 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:35.162307    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:35.164771 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:35.164742    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p4x9l\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:35.339907 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:35.339875    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:17:35.611856 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:35.611804    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"]
Sep 17 18:17:35.613676 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:17:35.613651    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2665f22_12be_44a7_a8fd_678d918e6db3.slice/crio-b7f9dcfe9c34675d000e50b7a3df0bba6b86a1db47b1dea82541f07007e4fa31 WatchSource:0}: Error finding container b7f9dcfe9c34675d000e50b7a3df0bba6b86a1db47b1dea82541f07007e4fa31: Status 404 returned error can't find the container with id b7f9dcfe9c34675d000e50b7a3df0bba6b86a1db47b1dea82541f07007e4fa31
Sep 17 18:17:35.778526 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:35.778489    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerStarted","Data":"af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169"}
Sep 17 18:17:35.778691 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:35.778545    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerStarted","Data":"b7f9dcfe9c34675d000e50b7a3df0bba6b86a1db47b1dea82541f07007e4fa31"}
Sep 17 18:17:39.214309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:39.214272    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:17:39.799324 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:39.799296    2585 generic.go:356] "Generic (PLEG): container finished" podID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerID="af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169" exitCode=0
Sep 17 18:17:39.799444 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:39.799340    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerDied","Data":"af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169"}
Sep 17 18:17:44.214288 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:44.214245    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:17:44.748427 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:44.748389    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:17:45.832499 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.832464    2585 generic.go:356] "Generic (PLEG): container finished" podID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerID="768be3ce3c4910f66738ac515b5114523cde99c11af88ccbb2ad54c520e46489" exitCode=137
Sep 17 18:17:45.832938 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.832551    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerDied","Data":"768be3ce3c4910f66738ac515b5114523cde99c11af88ccbb2ad54c520e46489"}
Sep 17 18:17:45.940922 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.940886    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:17:45.952768 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.952739    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls\") pod \"f3dd24f6-d921-40ae-98c2-4116c1533885\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") "
Sep 17 18:17:45.952931 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.952787    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-rbac-proxy-sar-config\") pod \"f3dd24f6-d921-40ae-98c2-4116c1533885\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") "
Sep 17 18:17:45.952931 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.952880    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmlfn\" (UniqueName: \"kubernetes.io/projected/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-api-access-kmlfn\") pod \"f3dd24f6-d921-40ae-98c2-4116c1533885\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") "
Sep 17 18:17:45.952931 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.952903    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-cabundle-cert\") pod \"f3dd24f6-d921-40ae-98c2-4116c1533885\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") "
Sep 17 18:17:45.953085 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.952991    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3dd24f6-d921-40ae-98c2-4116c1533885-kserve-provision-location\") pod \"f3dd24f6-d921-40ae-98c2-4116c1533885\" (UID: \"f3dd24f6-d921-40ae-98c2-4116c1533885\") "
Sep 17 18:17:45.953176 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.953135    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3dd24f6-d921-40ae-98c2-4116c1533885" (UID: "f3dd24f6-d921-40ae-98c2-4116c1533885"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:45.953327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.953310    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.953390 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.953370    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3dd24f6-d921-40ae-98c2-4116c1533885" (UID: "f3dd24f6-d921-40ae-98c2-4116c1533885"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:45.953442 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.953384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dd24f6-d921-40ae-98c2-4116c1533885-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3dd24f6-d921-40ae-98c2-4116c1533885" (UID: "f3dd24f6-d921-40ae-98c2-4116c1533885"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:45.955211 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.955181    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3dd24f6-d921-40ae-98c2-4116c1533885" (UID: "f3dd24f6-d921-40ae-98c2-4116c1533885"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:45.955467 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:45.955451    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-api-access-kmlfn" (OuterVolumeSpecName: "kube-api-access-kmlfn") pod "f3dd24f6-d921-40ae-98c2-4116c1533885" (UID: "f3dd24f6-d921-40ae-98c2-4116c1533885"). InnerVolumeSpecName "kube-api-access-kmlfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:46.054393 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.054348    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3dd24f6-d921-40ae-98c2-4116c1533885-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:46.054393 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.054392    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3dd24f6-d921-40ae-98c2-4116c1533885-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:46.054617 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.054408    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmlfn\" (UniqueName: \"kubernetes.io/projected/f3dd24f6-d921-40ae-98c2-4116c1533885-kube-api-access-kmlfn\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:46.054617 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.054421    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3dd24f6-d921-40ae-98c2-4116c1533885-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:17:46.841376 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.841347    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"
Sep 17 18:17:46.842099 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.841345    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8" event={"ID":"f3dd24f6-d921-40ae-98c2-4116c1533885","Type":"ContainerDied","Data":"d5b5ab08332024e39a4ebbb5c70e1d385d078c2c7f2cbc116b0cb8688b9fd281"}
Sep 17 18:17:46.842099 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.841508    2585 scope.go:117] "RemoveContainer" containerID="c6a4551b7b9cd432943c45e29491027ac62429c75e3c7f3cf567b0a8d84f3422"
Sep 17 18:17:46.853133 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.853117    2585 scope.go:117] "RemoveContainer" containerID="768be3ce3c4910f66738ac515b5114523cde99c11af88ccbb2ad54c520e46489"
Sep 17 18:17:46.862237 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.862214    2585 scope.go:117] "RemoveContainer" containerID="deff4fbb6a054e1fba1031707358c2002dc73a1f9f9c5029c48a2368f92510ac"
Sep 17 18:17:46.865951 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.865930    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"]
Sep 17 18:17:46.870362 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.870341    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-j84g8"]
Sep 17 18:17:46.961653 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:46.961621    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" path="/var/lib/kubelet/pods/f3dd24f6-d921-40ae-98c2-4116c1533885/volumes"
Sep 17 18:17:54.748941 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:17:54.748901    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:18:00.918926 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:00.918894    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerStarted","Data":"7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110"}
Sep 17 18:18:00.919358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:00.918979    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerStarted","Data":"323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652"}
Sep 17 18:18:00.919358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:00.919302    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:18:00.919468 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:00.919433    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:18:00.920860 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:00.920835    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:00.940590 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:00.940544    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podStartSLOduration=2.652999497 podStartE2EDuration="26.940530058s" totalImagesPullingTime="20.228310651s" totalInitContainerRuntime="4.05921991s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:00.938284352 +0000 UTC m=+2214.129622063" watchObservedRunningTime="2026-09-17 18:18:00.940530058 +0000 UTC m=+2214.131867743"
Sep 17 18:18:01.923756 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:01.923721    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:02.928432 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:02.928388    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:04.748906 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:04.748869    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:18:07.933851 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:07.933797    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:18:07.934440 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:07.934396    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:14.748935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:14.748893    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:18:17.935137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:17.935098    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:24.748593 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:24.748493    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:18:27.934480 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:27.934437    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:34.749248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:34.749210    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:18:37.934983 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:37.934938    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:39.954513 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:39.954473    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:18:47.935054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:47.935013    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:49.954971 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:49.954942    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:18:51.019507 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:51.019473    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"]
Sep 17 18:18:51.020036 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:51.019747    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container" containerID="cri-o://932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d" gracePeriod=30
Sep 17 18:18:51.020036 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:51.019824    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kube-rbac-proxy" containerID="cri-o://aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f" gracePeriod=30
Sep 17 18:18:51.158658 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:51.158628    2585 generic.go:356] "Generic (PLEG): container finished" podID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerID="aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f" exitCode=2
Sep 17 18:18:51.158802 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:51.158708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerDied","Data":"aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f"}
Sep 17 18:18:54.744765 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:54.744722    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 17 18:18:55.355574 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.355552    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:18:55.445156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445077    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-rbac-proxy-sar-config\") pod \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") "
Sep 17 18:18:55.445156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445144    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4bbz\" (UniqueName: \"kubernetes.io/projected/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-api-access-r4bbz\") pod \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") "
Sep 17 18:18:55.445384 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445169    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-cabundle-cert\") pod \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") "
Sep 17 18:18:55.445384 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445203    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kserve-provision-location\") pod \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") "
Sep 17 18:18:55.445384 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445332    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-proxy-tls\") pod \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\" (UID: \"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f\") "
Sep 17 18:18:55.445549 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445469    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" (UID: "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:55.445549 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445515    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" (UID: "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:55.445633 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445550    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" (UID: "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:55.445670 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445650    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.445705 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445668    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.445705 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.445681    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.447527 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.447506    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" (UID: "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:55.447731 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.447710    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-api-access-r4bbz" (OuterVolumeSpecName: "kube-api-access-r4bbz") pod "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" (UID: "c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f"). InnerVolumeSpecName "kube-api-access-r4bbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:55.546121 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.546088    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4bbz\" (UniqueName: \"kubernetes.io/projected/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-kube-api-access-r4bbz\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.546121 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:55.546119    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:18:56.185801 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.185770    2585 generic.go:356] "Generic (PLEG): container finished" podID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerID="932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d" exitCode=0
Sep 17 18:18:56.186255 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.185842    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerDied","Data":"932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d"}
Sep 17 18:18:56.186255 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.185870    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"
Sep 17 18:18:56.186255 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.185907    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w" event={"ID":"c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f","Type":"ContainerDied","Data":"b09599d2535a9ee60797d483669f3870ba0be2ee4044cb3226a7dc1e66e2b399"}
Sep 17 18:18:56.186255 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.185954    2585 scope.go:117] "RemoveContainer" containerID="aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f"
Sep 17 18:18:56.204446 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.203073    2585 scope.go:117] "RemoveContainer" containerID="932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d"
Sep 17 18:18:56.217789 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.217765    2585 scope.go:117] "RemoveContainer" containerID="fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee"
Sep 17 18:18:56.223217 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.223192    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"]
Sep 17 18:18:56.227637 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.227616    2585 scope.go:117] "RemoveContainer" containerID="aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f"
Sep 17 18:18:56.228004 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:18:56.227978    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f\": container with ID starting with aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f not found: ID does not exist" containerID="aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f"
Sep 17 18:18:56.228083 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.228016    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f"} err="failed to get container status \"aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f\": rpc error: code = NotFound desc = could not find container \"aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f\": container with ID starting with aae4af299123deb3016a5e77dce6c0639939de65cbb7ee50631565fe4fc7493f not found: ID does not exist"
Sep 17 18:18:56.228083 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.228042    2585 scope.go:117] "RemoveContainer" containerID="932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d"
Sep 17 18:18:56.228353 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:18:56.228333    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d\": container with ID starting with 932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d not found: ID does not exist" containerID="932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d"
Sep 17 18:18:56.228417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.228360    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d"} err="failed to get container status \"932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d\": rpc error: code = NotFound desc = could not find container \"932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d\": container with ID starting with 932be0a10914297b54773b9a1fd169ec65592b473527e2f328644aab9cd6e22d not found: ID does not exist"
Sep 17 18:18:56.228417 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.228381    2585 scope.go:117] "RemoveContainer" containerID="fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee"
Sep 17 18:18:56.228653 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:18:56.228621    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee\": container with ID starting with fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee not found: ID does not exist" containerID="fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee"
Sep 17 18:18:56.228759 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.228653    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee"} err="failed to get container status \"fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee\": rpc error: code = NotFound desc = could not find container \"fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee\": container with ID starting with fe02d75b1296a6f422c403695b05172adb839c5ad1c465ab8b41f52986a468ee not found: ID does not exist"
Sep 17 18:18:56.232079 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.232042    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-5bh2w"]
Sep 17 18:18:56.278300 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278270    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"]
Sep 17 18:18:56.278869 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278853    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="storage-initializer"
Sep 17 18:18:56.278918 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278873    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="storage-initializer"
Sep 17 18:18:56.278918 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278887    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy"
Sep 17 18:18:56.278918 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278895    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy"
Sep 17 18:18:56.278918 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278909    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kube-rbac-proxy"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278919    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kube-rbac-proxy"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278935    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278943    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278954    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="storage-initializer"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278962    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="storage-initializer"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278983    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container"
Sep 17 18:18:56.279041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.278991    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container"
Sep 17 18:18:56.279247 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.279070    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kube-rbac-proxy"
Sep 17 18:18:56.279247 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.279083    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kserve-container"
Sep 17 18:18:56.279247 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.279092    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3dd24f6-d921-40ae-98c2-4116c1533885" containerName="kserve-container"
Sep 17 18:18:56.279247 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.279104    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" containerName="kube-rbac-proxy"
Sep 17 18:18:56.281561 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.281545    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.284257 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.284235    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:18:56.284483 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.284277    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:18:56.291091 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.291070    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"]
Sep 17 18:18:56.353905 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.353879    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875e02fa-930a-4649-8863-746ec8707b61-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.354031 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.353909    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.354031 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.353934    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.354104 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.354025    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.354104 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.354072    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfz8m\" (UniqueName: \"kubernetes.io/projected/875e02fa-930a-4649-8863-746ec8707b61-kube-api-access-sfz8m\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.455354 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.455288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875e02fa-930a-4649-8863-746ec8707b61-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.455354 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.455323    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.455354 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.455350    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.455578 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.455382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.455578 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.455425    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfz8m\" (UniqueName: \"kubernetes.io/projected/875e02fa-930a-4649-8863-746ec8707b61-kube-api-access-sfz8m\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.455688 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:18:56.455584    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:18:56.455688 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:18:56.455651    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls podName:875e02fa-930a-4649-8863-746ec8707b61 nodeName:}" failed. No retries permitted until 2026-09-17 18:18:56.955631554 +0000 UTC m=+2270.146969223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls") pod "isvc-canary-stop-predictor-646f5bdbcb-2578m" (UID: "875e02fa-930a-4649-8863-746ec8707b61") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:18:56.455800 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.455714    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875e02fa-930a-4649-8863-746ec8707b61-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.456148 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.456126    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.456201 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.456131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.463377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.463357    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfz8m\" (UniqueName: \"kubernetes.io/projected/875e02fa-930a-4649-8863-746ec8707b61-kube-api-access-sfz8m\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.959367 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.959330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:56.960881 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.960851    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f" path="/var/lib/kubelet/pods/c1e7ce65-a8b1-4c8c-862e-17b5c9f24c4f/volumes"
Sep 17 18:18:56.961803 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:56.961776    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-2578m\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:57.193592 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:57.193567    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:18:57.464865 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:57.464842    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"]
Sep 17 18:18:57.466944 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:18:57.466918    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875e02fa_930a_4649_8863_746ec8707b61.slice/crio-c616dd9276923a88622e51ffe92bf0c430a8d9b003b3b7f4c8e4749b85014f8e WatchSource:0}: Error finding container c616dd9276923a88622e51ffe92bf0c430a8d9b003b3b7f4c8e4749b85014f8e: Status 404 returned error can't find the container with id c616dd9276923a88622e51ffe92bf0c430a8d9b003b3b7f4c8e4749b85014f8e
Sep 17 18:18:57.934705 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:57.934663    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:18:58.199108 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:58.199017    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerStarted","Data":"d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345"}
Sep 17 18:18:58.199108 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:18:58.199076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerStarted","Data":"c616dd9276923a88622e51ffe92bf0c430a8d9b003b3b7f4c8e4749b85014f8e"}
Sep 17 18:19:01.215000 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:01.214969    2585 generic.go:356] "Generic (PLEG): container finished" podID="875e02fa-930a-4649-8863-746ec8707b61" containerID="d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345" exitCode=0
Sep 17 18:19:01.215340 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:01.215044    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerDied","Data":"d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345"}
Sep 17 18:19:02.227619 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:02.227581    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerStarted","Data":"a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67"}
Sep 17 18:19:02.228012 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:02.227671    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerStarted","Data":"7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24"}
Sep 17 18:19:02.228012 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:02.227953    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:19:02.248041 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:02.247996    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podStartSLOduration=2.655568038 podStartE2EDuration="6.247983372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.592415334s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:02.245998475 +0000 UTC m=+2275.437336159" watchObservedRunningTime="2026-09-17 18:19:02.247983372 +0000 UTC m=+2275.439321056"
Sep 17 18:19:03.232198 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:03.232164    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:19:03.233431 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:03.233406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:04.236777 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:04.236737    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:07.935608 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:07.935578    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:19:09.241025 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:09.240995    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:19:09.241440 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:09.241415    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:14.904599 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:14.904569    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"]
Sep 17 18:19:14.905036 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:14.904794    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" containerID="cri-o://323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652" gracePeriod=30
Sep 17 18:19:14.905036 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:14.904852    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kube-rbac-proxy" containerID="cri-o://7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110" gracePeriod=30
Sep 17 18:19:15.288674 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:15.288596    2585 generic.go:356] "Generic (PLEG): container finished" podID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerID="7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110" exitCode=2
Sep 17 18:19:15.288846 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:15.288668    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerDied","Data":"7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110"}
Sep 17 18:19:17.929244 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:17.929196    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 17 18:19:17.934710 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:17.934685    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:19:18.839016 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.838992    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:19:18.937203 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937172    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-rbac-proxy-sar-config\") pod \"f2665f22-12be-44a7-a8fd-678d918e6db3\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") "
Sep 17 18:19:18.937581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937208    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-cabundle-cert\") pod \"f2665f22-12be-44a7-a8fd-678d918e6db3\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") "
Sep 17 18:19:18.937581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937305    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls\") pod \"f2665f22-12be-44a7-a8fd-678d918e6db3\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") "
Sep 17 18:19:18.937581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937339    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4psj\" (UniqueName: \"kubernetes.io/projected/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-api-access-d4psj\") pod \"f2665f22-12be-44a7-a8fd-678d918e6db3\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") "
Sep 17 18:19:18.937581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937484    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2665f22-12be-44a7-a8fd-678d918e6db3" (UID: "f2665f22-12be-44a7-a8fd-678d918e6db3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:18.937581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937487    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2665f22-12be-44a7-a8fd-678d918e6db3-kserve-provision-location\") pod \"f2665f22-12be-44a7-a8fd-678d918e6db3\" (UID: \"f2665f22-12be-44a7-a8fd-678d918e6db3\") "
Sep 17 18:19:18.937581 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937494    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2665f22-12be-44a7-a8fd-678d918e6db3" (UID: "f2665f22-12be-44a7-a8fd-678d918e6db3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:18.937912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937752    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2665f22-12be-44a7-a8fd-678d918e6db3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2665f22-12be-44a7-a8fd-678d918e6db3" (UID: "f2665f22-12be-44a7-a8fd-678d918e6db3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:18.937912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937836    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:19:18.937912 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.937858    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2665f22-12be-44a7-a8fd-678d918e6db3-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:19:18.939641 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.939580    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2665f22-12be-44a7-a8fd-678d918e6db3" (UID: "f2665f22-12be-44a7-a8fd-678d918e6db3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:18.939641 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:18.939611    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-api-access-d4psj" (OuterVolumeSpecName: "kube-api-access-d4psj") pod "f2665f22-12be-44a7-a8fd-678d918e6db3" (UID: "f2665f22-12be-44a7-a8fd-678d918e6db3"). InnerVolumeSpecName "kube-api-access-d4psj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:19.038788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.038764    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2665f22-12be-44a7-a8fd-678d918e6db3-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:19:19.038788 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.038785    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4psj\" (UniqueName: \"kubernetes.io/projected/f2665f22-12be-44a7-a8fd-678d918e6db3-kube-api-access-d4psj\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:19:19.038960 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.038795    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2665f22-12be-44a7-a8fd-678d918e6db3-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:19:19.242378 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.242285    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:19.311054 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.311021    2585 generic.go:356] "Generic (PLEG): container finished" podID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerID="323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652" exitCode=0
Sep 17 18:19:19.311206 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.311074    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerDied","Data":"323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652"}
Sep 17 18:19:19.311206 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.311120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l" event={"ID":"f2665f22-12be-44a7-a8fd-678d918e6db3","Type":"ContainerDied","Data":"b7f9dcfe9c34675d000e50b7a3df0bba6b86a1db47b1dea82541f07007e4fa31"}
Sep 17 18:19:19.311206 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.311144    2585 scope.go:117] "RemoveContainer" containerID="7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110"
Sep 17 18:19:19.311206 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.311075    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"
Sep 17 18:19:19.319718 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.319703    2585 scope.go:117] "RemoveContainer" containerID="323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652"
Sep 17 18:19:19.327703 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.327687    2585 scope.go:117] "RemoveContainer" containerID="af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169"
Sep 17 18:19:19.329664 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.329643    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"]
Sep 17 18:19:19.332415 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.332391    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p4x9l"]
Sep 17 18:19:19.336040 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.336022    2585 scope.go:117] "RemoveContainer" containerID="7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110"
Sep 17 18:19:19.336310 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:19:19.336289    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110\": container with ID starting with 7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110 not found: ID does not exist" containerID="7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110"
Sep 17 18:19:19.336355 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.336324    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110"} err="failed to get container status \"7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110\": rpc error: code = NotFound desc = could not find container \"7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110\": container with ID starting with 7830f7a22a30131207606de9788c4d57e45b4d1e056112641a1721257856d110 not found: ID does not exist"
Sep 17 18:19:19.336355 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.336342    2585 scope.go:117] "RemoveContainer" containerID="323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652"
Sep 17 18:19:19.336583 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:19:19.336565    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652\": container with ID starting with 323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652 not found: ID does not exist" containerID="323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652"
Sep 17 18:19:19.336628 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.336589    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652"} err="failed to get container status \"323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652\": rpc error: code = NotFound desc = could not find container \"323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652\": container with ID starting with 323e453a606df1d9ed942356db7f8594c1914ed131555880b1f70db32561e652 not found: ID does not exist"
Sep 17 18:19:19.336628 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.336605    2585 scope.go:117] "RemoveContainer" containerID="af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169"
Sep 17 18:19:19.336859 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:19:19.336844    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169\": container with ID starting with af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169 not found: ID does not exist" containerID="af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169"
Sep 17 18:19:19.336904 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:19.336865    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169"} err="failed to get container status \"af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169\": rpc error: code = NotFound desc = could not find container \"af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169\": container with ID starting with af72b0f9d331f37125d61be1ed862b54ab60212c2e52f1fa0456fc13fbf71169 not found: ID does not exist"
Sep 17 18:19:20.960090 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:20.960051    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" path="/var/lib/kubelet/pods/f2665f22-12be-44a7-a8fd-678d918e6db3/volumes"
Sep 17 18:19:29.242046 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:29.242007    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:39.242094 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:39.242050    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:49.241940 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:49.241860    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:19:59.244154 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:19:59.243954    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:20:09.241981 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:09.241933    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:20:16.959512 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:16.959481    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:20:26.227765 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:26.227731    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"]
Sep 17 18:20:26.228165 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:26.228025    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" containerID="cri-o://7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24" gracePeriod=30
Sep 17 18:20:26.228165 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:26.228101    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kube-rbac-proxy" containerID="cri-o://a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67" gracePeriod=30
Sep 17 18:20:26.631786 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:26.631755    2585 generic.go:356] "Generic (PLEG): container finished" podID="875e02fa-930a-4649-8863-746ec8707b61" containerID="a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67" exitCode=2
Sep 17 18:20:26.631966 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:26.631829    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerDied","Data":"a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67"}
Sep 17 18:20:26.954678 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:26.954588    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:20:29.236969 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:29.236924    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 17 18:20:30.571154 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.571132    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:20:30.653404 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.653333    2585 generic.go:356] "Generic (PLEG): container finished" podID="875e02fa-930a-4649-8863-746ec8707b61" containerID="7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24" exitCode=0
Sep 17 18:20:30.653404 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.653386    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"
Sep 17 18:20:30.653573 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.653390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerDied","Data":"7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24"}
Sep 17 18:20:30.653573 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.653510    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m" event={"ID":"875e02fa-930a-4649-8863-746ec8707b61","Type":"ContainerDied","Data":"c616dd9276923a88622e51ffe92bf0c430a8d9b003b3b7f4c8e4749b85014f8e"}
Sep 17 18:20:30.653573 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.653536    2585 scope.go:117] "RemoveContainer" containerID="a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67"
Sep 17 18:20:30.662026 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.662009    2585 scope.go:117] "RemoveContainer" containerID="7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24"
Sep 17 18:20:30.669528 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.669506    2585 scope.go:117] "RemoveContainer" containerID="d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345"
Sep 17 18:20:30.676603 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.676586    2585 scope.go:117] "RemoveContainer" containerID="a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67"
Sep 17 18:20:30.676887 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:20:30.676861    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67\": container with ID starting with a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67 not found: ID does not exist" containerID="a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67"
Sep 17 18:20:30.676970 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.676891    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67"} err="failed to get container status \"a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67\": rpc error: code = NotFound desc = could not find container \"a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67\": container with ID starting with a8b81b6cc0ebca05b307e3b04d4630dc9dbd83e13f9203c2dd84d825b9820d67 not found: ID does not exist"
Sep 17 18:20:30.676970 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.676907    2585 scope.go:117] "RemoveContainer" containerID="7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24"
Sep 17 18:20:30.677136 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:20:30.677120    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24\": container with ID starting with 7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24 not found: ID does not exist" containerID="7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24"
Sep 17 18:20:30.677226 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.677138    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24"} err="failed to get container status \"7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24\": rpc error: code = NotFound desc = could not find container \"7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24\": container with ID starting with 7b92af551c2a83fef7fa48b7df419a65ded2dc41517790ed70d712b032fc6b24 not found: ID does not exist"
Sep 17 18:20:30.677226 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.677150    2585 scope.go:117] "RemoveContainer" containerID="d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345"
Sep 17 18:20:30.677404 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:20:30.677386    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345\": container with ID starting with d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345 not found: ID does not exist" containerID="d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345"
Sep 17 18:20:30.677451 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.677413    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345"} err="failed to get container status \"d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345\": rpc error: code = NotFound desc = could not find container \"d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345\": container with ID starting with d14231e4ace2540f6983410886a1d1020732c8b58758947debc69dc04d9ef345 not found: ID does not exist"
Sep 17 18:20:30.724676 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.724653    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-kube-rbac-proxy-sar-config\") pod \"875e02fa-930a-4649-8863-746ec8707b61\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") "
Sep 17 18:20:30.724779 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.724715    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfz8m\" (UniqueName: \"kubernetes.io/projected/875e02fa-930a-4649-8863-746ec8707b61-kube-api-access-sfz8m\") pod \"875e02fa-930a-4649-8863-746ec8707b61\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") "
Sep 17 18:20:30.724779 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.724748    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875e02fa-930a-4649-8863-746ec8707b61-kserve-provision-location\") pod \"875e02fa-930a-4649-8863-746ec8707b61\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") "
Sep 17 18:20:30.724933 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.724804    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-cabundle-cert\") pod \"875e02fa-930a-4649-8863-746ec8707b61\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") "
Sep 17 18:20:30.724933 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.724865    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls\") pod \"875e02fa-930a-4649-8863-746ec8707b61\" (UID: \"875e02fa-930a-4649-8863-746ec8707b61\") "
Sep 17 18:20:30.725138 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.725117    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "875e02fa-930a-4649-8863-746ec8707b61" (UID: "875e02fa-930a-4649-8863-746ec8707b61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:30.725207 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.725151    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875e02fa-930a-4649-8863-746ec8707b61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "875e02fa-930a-4649-8863-746ec8707b61" (UID: "875e02fa-930a-4649-8863-746ec8707b61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:30.725270 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.725251    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "875e02fa-930a-4649-8863-746ec8707b61" (UID: "875e02fa-930a-4649-8863-746ec8707b61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:30.726757 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.726727    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875e02fa-930a-4649-8863-746ec8707b61-kube-api-access-sfz8m" (OuterVolumeSpecName: "kube-api-access-sfz8m") pod "875e02fa-930a-4649-8863-746ec8707b61" (UID: "875e02fa-930a-4649-8863-746ec8707b61"). InnerVolumeSpecName "kube-api-access-sfz8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:30.726908 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.726887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "875e02fa-930a-4649-8863-746ec8707b61" (UID: "875e02fa-930a-4649-8863-746ec8707b61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:30.826223 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.826200    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfz8m\" (UniqueName: \"kubernetes.io/projected/875e02fa-930a-4649-8863-746ec8707b61-kube-api-access-sfz8m\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.826223 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.826221    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875e02fa-930a-4649-8863-746ec8707b61-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.826345 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.826230    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.826345 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.826239    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/875e02fa-930a-4649-8863-746ec8707b61-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.826345 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.826250    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/875e02fa-930a-4649-8863-746ec8707b61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.977752 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.977721    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"]
Sep 17 18:20:30.981138 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:30.981114    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-2578m"]
Sep 17 18:20:32.959524 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:20:32.959490    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="875e02fa-930a-4649-8863-746ec8707b61" path="/var/lib/kubelet/pods/875e02fa-930a-4649-8863-746ec8707b61/volumes"
Sep 17 18:22:17.573409 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:22:17.573377    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:22:17.576058 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:22:17.576034    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:26:57.576660 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.576627    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"]
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.576980    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="storage-initializer"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.576992    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="storage-initializer"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577002    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kube-rbac-proxy"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577008    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kube-rbac-proxy"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577020    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kube-rbac-proxy"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577026    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kube-rbac-proxy"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577035    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577040    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577052    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577057    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577080    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="storage-initializer"
Sep 17 18:26:57.577120 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577085    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="storage-initializer"
Sep 17 18:26:57.577495 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577135    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kserve-container"
Sep 17 18:26:57.577495 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577145    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2665f22-12be-44a7-a8fd-678d918e6db3" containerName="kube-rbac-proxy"
Sep 17 18:26:57.577495 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577153    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kube-rbac-proxy"
Sep 17 18:26:57.577495 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.577160    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="875e02fa-930a-4649-8863-746ec8707b61" containerName="kserve-container"
Sep 17 18:26:57.580280 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.580263    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.583046 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583021    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:26:57.583046 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583040    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:26:57.583249 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583040    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 18:26:57.583249 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583105    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:26:57.583249 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583141    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:26:57.583249 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583216    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:26:57.583475 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.583316    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:26:57.598617 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.598594    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"]
Sep 17 18:26:57.673032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.673000    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.673162 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.673071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.673162 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.673134    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.673256 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.673213    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndk5t\" (UniqueName: \"kubernetes.io/projected/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-api-access-ndk5t\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.673256 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.673247    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.773948 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.773922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774102 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.773958    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774102 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.774008    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774102 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.774044    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndk5t\" (UniqueName: \"kubernetes.io/projected/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-api-access-ndk5t\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774102 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.774071    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774102 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:26:57.774079    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 18:26:57.774376 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:26:57.774172    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls podName:da5f8454-67f7-4f7f-8dc9-9a438428d85a nodeName:}" failed. No retries permitted until 2026-09-17 18:26:58.274148391 +0000 UTC m=+2751.465486056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" (UID: "da5f8454-67f7-4f7f-8dc9-9a438428d85a") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 18:26:57.774492 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.774470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.774724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.774838 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.774789    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:57.782674 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:57.782656    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndk5t\" (UniqueName: \"kubernetes.io/projected/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-api-access-ndk5t\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:58.277898 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:58.277859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:58.280441 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:58.280417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:58.491433 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:58.491396    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:26:58.761134 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:58.761110    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"]
Sep 17 18:26:58.763870 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:26:58.763848    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5f8454_67f7_4f7f_8dc9_9a438428d85a.slice/crio-99b456a6eafcd21de7ea5e8a0d3093ff24fc558fc62e1c476d467f4752b5f7e6 WatchSource:0}: Error finding container 99b456a6eafcd21de7ea5e8a0d3093ff24fc558fc62e1c476d467f4752b5f7e6: Status 404 returned error can't find the container with id 99b456a6eafcd21de7ea5e8a0d3093ff24fc558fc62e1c476d467f4752b5f7e6
Sep 17 18:26:59.409286 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:59.409244    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerStarted","Data":"7b452f7ea806802f3ef114ee680dc9328cef2f703802635b36da457f64aa1bd6"}
Sep 17 18:26:59.409434 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:26:59.409321    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerStarted","Data":"99b456a6eafcd21de7ea5e8a0d3093ff24fc558fc62e1c476d467f4752b5f7e6"}
Sep 17 18:27:00.415829 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:00.415783    2585 generic.go:356] "Generic (PLEG): container finished" podID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerID="7b452f7ea806802f3ef114ee680dc9328cef2f703802635b36da457f64aa1bd6" exitCode=0
Sep 17 18:27:00.416213 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:00.415874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerDied","Data":"7b452f7ea806802f3ef114ee680dc9328cef2f703802635b36da457f64aa1bd6"}
Sep 17 18:27:01.422605 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:01.422566    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerStarted","Data":"0032a1fd454f04d0fd859d4dda96e3c4a1088ac8ff0ac721c446e242497afd50"}
Sep 17 18:27:01.423027 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:01.422633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerStarted","Data":"5cfde61f68f21ac39f9bf4570bd6a03195c54f732d81afafa60f6c807c287123"}
Sep 17 18:27:01.423027 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:01.422667    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:27:01.423027 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:01.422696    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:27:01.424099 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:01.424074    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:01.442082 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:01.442034    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podStartSLOduration=3.302072629 podStartE2EDuration="4.442015322s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.139942693s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:01.439869147 +0000 UTC m=+2754.631206832" watchObservedRunningTime="2026-09-17 18:27:01.442015322 +0000 UTC m=+2754.633353204"
Sep 17 18:27:02.426770 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:02.426728    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:07.431263 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:07.431230    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:27:07.431846 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:07.431800    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:17.432225 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:17.432189    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:17.605432 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:17.605403    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:27:17.608390 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:17.608371    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:27:27.432619 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:27.432530    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:37.432520 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:37.432476    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:47.432233 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:47.432193    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:27:57.432085 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:27:57.432048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:28:07.432802 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:07.432750    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:28:17.432505 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:17.432472    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:28:27.599244 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:27.599211    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"]
Sep 17 18:28:27.599726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:27.599469    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container" containerID="cri-o://5cfde61f68f21ac39f9bf4570bd6a03195c54f732d81afafa60f6c807c287123" gracePeriod=30
Sep 17 18:28:27.599726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:27.599563    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kube-rbac-proxy" containerID="cri-o://0032a1fd454f04d0fd859d4dda96e3c4a1088ac8ff0ac721c446e242497afd50" gracePeriod=30
Sep 17 18:28:27.820873 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:27.820838    2585 generic.go:356] "Generic (PLEG): container finished" podID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerID="0032a1fd454f04d0fd859d4dda96e3c4a1088ac8ff0ac721c446e242497afd50" exitCode=2
Sep 17 18:28:27.821021 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:27.820881    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerDied","Data":"0032a1fd454f04d0fd859d4dda96e3c4a1088ac8ff0ac721c446e242497afd50"}
Sep 17 18:28:28.725647 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.725615    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"]
Sep 17 18:28:28.729579 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.729554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.732173 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.732153    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 18:28:28.732173 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.732174    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:28:28.741610 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.741583    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"]
Sep 17 18:28:28.793974 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.793941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.794132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.794044    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94eaf480-c4ce-409b-817e-11e80a942aaf-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.794132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.794084    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94eaf480-c4ce-409b-817e-11e80a942aaf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.794132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.794115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbf8l\" (UniqueName: \"kubernetes.io/projected/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-api-access-lbf8l\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.894971 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.894938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94eaf480-c4ce-409b-817e-11e80a942aaf-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.895151 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.894979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94eaf480-c4ce-409b-817e-11e80a942aaf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.895151 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.895006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbf8l\" (UniqueName: \"kubernetes.io/projected/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-api-access-lbf8l\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.895151 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.895052    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.895470 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.895443    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94eaf480-c4ce-409b-817e-11e80a942aaf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.896090 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.896065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.897717 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.897697    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94eaf480-c4ce-409b-817e-11e80a942aaf-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:28.903139 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:28.903116    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbf8l\" (UniqueName: \"kubernetes.io/projected/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-api-access-lbf8l\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:29.041027 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:29.040965    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:29.312821 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:29.312783    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"]
Sep 17 18:28:29.314510 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:28:29.314482    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94eaf480_c4ce_409b_817e_11e80a942aaf.slice/crio-c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5 WatchSource:0}: Error finding container c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5: Status 404 returned error can't find the container with id c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5
Sep 17 18:28:29.832990 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:29.832960    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" event={"ID":"94eaf480-c4ce-409b-817e-11e80a942aaf","Type":"ContainerStarted","Data":"dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706"}
Sep 17 18:28:29.833358 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:29.833021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" event={"ID":"94eaf480-c4ce-409b-817e-11e80a942aaf","Type":"ContainerStarted","Data":"c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5"}
Sep 17 18:28:31.845080 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.845053    2585 generic.go:356] "Generic (PLEG): container finished" podID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerID="5cfde61f68f21ac39f9bf4570bd6a03195c54f732d81afafa60f6c807c287123" exitCode=0
Sep 17 18:28:31.845384 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.845127    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerDied","Data":"5cfde61f68f21ac39f9bf4570bd6a03195c54f732d81afafa60f6c807c287123"}
Sep 17 18:28:31.845384 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.845179    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg" event={"ID":"da5f8454-67f7-4f7f-8dc9-9a438428d85a","Type":"ContainerDied","Data":"99b456a6eafcd21de7ea5e8a0d3093ff24fc558fc62e1c476d467f4752b5f7e6"}
Sep 17 18:28:31.845384 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.845200    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b456a6eafcd21de7ea5e8a0d3093ff24fc558fc62e1c476d467f4752b5f7e6"
Sep 17 18:28:31.847137 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.847117    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:28:31.923110 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923078    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndk5t\" (UniqueName: \"kubernetes.io/projected/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-api-access-ndk5t\") pod \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") "
Sep 17 18:28:31.923277 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923148    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls\") pod \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") "
Sep 17 18:28:31.923277 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923171    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kserve-provision-location\") pod \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") "
Sep 17 18:28:31.923277 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923213    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-rbac-proxy-sar-config\") pod \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") "
Sep 17 18:28:31.923277 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923236    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-cabundle-cert\") pod \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\" (UID: \"da5f8454-67f7-4f7f-8dc9-9a438428d85a\") "
Sep 17 18:28:31.923564 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923535    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da5f8454-67f7-4f7f-8dc9-9a438428d85a" (UID: "da5f8454-67f7-4f7f-8dc9-9a438428d85a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:31.923637 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923568    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da5f8454-67f7-4f7f-8dc9-9a438428d85a" (UID: "da5f8454-67f7-4f7f-8dc9-9a438428d85a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:31.923637 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.923608    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da5f8454-67f7-4f7f-8dc9-9a438428d85a" (UID: "da5f8454-67f7-4f7f-8dc9-9a438428d85a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:31.925403 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.925369    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-api-access-ndk5t" (OuterVolumeSpecName: "kube-api-access-ndk5t") pod "da5f8454-67f7-4f7f-8dc9-9a438428d85a" (UID: "da5f8454-67f7-4f7f-8dc9-9a438428d85a"). InnerVolumeSpecName "kube-api-access-ndk5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:31.925516 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:31.925466    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da5f8454-67f7-4f7f-8dc9-9a438428d85a" (UID: "da5f8454-67f7-4f7f-8dc9-9a438428d85a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:32.023935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.023883    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da5f8454-67f7-4f7f-8dc9-9a438428d85a-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:32.023935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.023904    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:32.023935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.023914    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:32.023935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.023922    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8454-67f7-4f7f-8dc9-9a438428d85a-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:32.023935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.023931    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndk5t\" (UniqueName: \"kubernetes.io/projected/da5f8454-67f7-4f7f-8dc9-9a438428d85a-kube-api-access-ndk5t\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:32.851741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.851720    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/0.log"
Sep 17 18:28:32.852092 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.851754    2585 generic.go:356] "Generic (PLEG): container finished" podID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerID="dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706" exitCode=1
Sep 17 18:28:32.852092 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.851842    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" event={"ID":"94eaf480-c4ce-409b-817e-11e80a942aaf","Type":"ContainerDied","Data":"dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706"}
Sep 17 18:28:32.852092 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.851926    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"
Sep 17 18:28:32.905524 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.905503    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"]
Sep 17 18:28:32.911128 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.911107    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-b5tgg"]
Sep 17 18:28:32.960936 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:32.960901    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" path="/var/lib/kubelet/pods/da5f8454-67f7-4f7f-8dc9-9a438428d85a/volumes"
Sep 17 18:28:33.858552 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:33.858525    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/0.log"
Sep 17 18:28:33.858922 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:33.858590    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" event={"ID":"94eaf480-c4ce-409b-817e-11e80a942aaf","Type":"ContainerStarted","Data":"2a5647d9fdf3399cbf1d3f36bcadc3e84edad95e00a10b7de5c8c18c1f829cc3"}
Sep 17 18:28:36.876416 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.876392    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/1.log"
Sep 17 18:28:36.876841 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.876826    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/0.log"
Sep 17 18:28:36.876896 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.876860    2585 generic.go:356] "Generic (PLEG): container finished" podID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerID="2a5647d9fdf3399cbf1d3f36bcadc3e84edad95e00a10b7de5c8c18c1f829cc3" exitCode=1
Sep 17 18:28:36.876978 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.876950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" event={"ID":"94eaf480-c4ce-409b-817e-11e80a942aaf","Type":"ContainerDied","Data":"2a5647d9fdf3399cbf1d3f36bcadc3e84edad95e00a10b7de5c8c18c1f829cc3"}
Sep 17 18:28:36.877055 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.877043    2585 scope.go:117] "RemoveContainer" containerID="dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706"
Sep 17 18:28:36.877410 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.877396    2585 scope.go:117] "RemoveContainer" containerID="dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706"
Sep 17 18:28:36.887606 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:28:36.887585    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_kserve-ci-e2e-test(94eaf480-c4ce-409b-817e-11e80a942aaf)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf"
Sep 17 18:28:36.887706 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:28:36.887680    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_kserve-ci-e2e-test_94eaf480-c4ce-409b-817e-11e80a942aaf_0 in pod sandbox c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5 from index: no such id: 'dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706'" containerID="dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706"
Sep 17 18:28:36.887752 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:36.887719    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_kserve-ci-e2e-test_94eaf480-c4ce-409b-817e-11e80a942aaf_0 in pod sandbox c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5 from index: no such id: 'dd05492d3a48b6973ff13fb9298c7cc2d3afd778cfd0a09d15cf37d61ba6e706'"
Sep 17 18:28:37.883517 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:37.883489    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/1.log"
Sep 17 18:28:38.700330 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.700296    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"]
Sep 17 18:28:38.834285 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.834264    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/1.log"
Sep 17 18:28:38.836632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.836612    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:38.871434 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871407    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94eaf480-c4ce-409b-817e-11e80a942aaf-kserve-provision-location\") pod \"94eaf480-c4ce-409b-817e-11e80a942aaf\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") "
Sep 17 18:28:38.871547 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871469    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-rbac-proxy-sar-config\") pod \"94eaf480-c4ce-409b-817e-11e80a942aaf\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") "
Sep 17 18:28:38.871547 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871511    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94eaf480-c4ce-409b-817e-11e80a942aaf-proxy-tls\") pod \"94eaf480-c4ce-409b-817e-11e80a942aaf\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") "
Sep 17 18:28:38.871665 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871591    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbf8l\" (UniqueName: \"kubernetes.io/projected/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-api-access-lbf8l\") pod \"94eaf480-c4ce-409b-817e-11e80a942aaf\" (UID: \"94eaf480-c4ce-409b-817e-11e80a942aaf\") "
Sep 17 18:28:38.871721 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871666    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94eaf480-c4ce-409b-817e-11e80a942aaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94eaf480-c4ce-409b-817e-11e80a942aaf" (UID: "94eaf480-c4ce-409b-817e-11e80a942aaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:38.871871 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871848    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94eaf480-c4ce-409b-817e-11e80a942aaf" (UID: "94eaf480-c4ce-409b-817e-11e80a942aaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:38.871924 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.871866    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94eaf480-c4ce-409b-817e-11e80a942aaf-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:38.873711 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.873690    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94eaf480-c4ce-409b-817e-11e80a942aaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94eaf480-c4ce-409b-817e-11e80a942aaf" (UID: "94eaf480-c4ce-409b-817e-11e80a942aaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:38.873853 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.873833    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-api-access-lbf8l" (OuterVolumeSpecName: "kube-api-access-lbf8l") pod "94eaf480-c4ce-409b-817e-11e80a942aaf" (UID: "94eaf480-c4ce-409b-817e-11e80a942aaf"). InnerVolumeSpecName "kube-api-access-lbf8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:38.888919 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.888903    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj_94eaf480-c4ce-409b-817e-11e80a942aaf/storage-initializer/1.log"
Sep 17 18:28:38.889224 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.888988    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"
Sep 17 18:28:38.889224 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.889021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj" event={"ID":"94eaf480-c4ce-409b-817e-11e80a942aaf","Type":"ContainerDied","Data":"c1fc24b14296012830b8d877ad8d5dfc81925f3b27591716c634d82726c5bbc5"}
Sep 17 18:28:38.889224 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.889092    2585 scope.go:117] "RemoveContainer" containerID="2a5647d9fdf3399cbf1d3f36bcadc3e84edad95e00a10b7de5c8c18c1f829cc3"
Sep 17 18:28:38.924858 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.924825    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"]
Sep 17 18:28:38.928013 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.927993    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-smrpj"]
Sep 17 18:28:38.966246 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.966178    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" path="/var/lib/kubelet/pods/94eaf480-c4ce-409b-817e-11e80a942aaf/volumes"
Sep 17 18:28:38.972585 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.972564    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:38.972673 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.972586    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94eaf480-c4ce-409b-817e-11e80a942aaf-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:38.972673 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:38.972597    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbf8l\" (UniqueName: \"kubernetes.io/projected/94eaf480-c4ce-409b-817e-11e80a942aaf-kube-api-access-lbf8l\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:28:39.833169 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833129    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"]
Sep 17 18:28:39.833535 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833518    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerName="storage-initializer"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833538    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerName="storage-initializer"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833561    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kube-rbac-proxy"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833570    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kube-rbac-proxy"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833581    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833591    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833602    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="storage-initializer"
Sep 17 18:28:39.833632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833610    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="storage-initializer"
Sep 17 18:28:39.834032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833734    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kserve-container"
Sep 17 18:28:39.834032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833748    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerName="storage-initializer"
Sep 17 18:28:39.834032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833762    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerName="storage-initializer"
Sep 17 18:28:39.834032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833772    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5f8454-67f7-4f7f-8dc9-9a438428d85a" containerName="kube-rbac-proxy"
Sep 17 18:28:39.834032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833902    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerName="storage-initializer"
Sep 17 18:28:39.834032 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.833913    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="94eaf480-c4ce-409b-817e-11e80a942aaf" containerName="storage-initializer"
Sep 17 18:28:39.838614 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.838594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.841650 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.841620    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:28:39.841650 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.841643    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:28:39.841858 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.841654    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:28:39.841858 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.841625    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 18:28:39.842775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.842745    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:28:39.842775 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.842750    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:28:39.842955 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.842822    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:28:39.847622 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.847599    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"]
Sep 17 18:28:39.881357 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.881329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.881441 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.881361    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d787c22-368f-4ebb-92f4-8d44601ef3ef-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.881498 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.881432    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8lj\" (UniqueName: \"kubernetes.io/projected/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-api-access-jq8lj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.881498 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.881470    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.881570 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.881503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.982839 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.982791    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.982839 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.982839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d787c22-368f-4ebb-92f4-8d44601ef3ef-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.983315 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.982879    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8lj\" (UniqueName: \"kubernetes.io/projected/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-api-access-jq8lj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.983315 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.982910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.983315 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.982931    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.983315 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.983308    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.983521 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.983503    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.983521 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.983514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.985649 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.985621    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d787c22-368f-4ebb-92f4-8d44601ef3ef-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:39.991119 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:39.991097    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8lj\" (UniqueName: \"kubernetes.io/projected/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-api-access-jq8lj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:40.149697 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:40.149670    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:40.423153 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:40.423126    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"]
Sep 17 18:28:40.425294 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:28:40.425262    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d787c22_368f_4ebb_92f4_8d44601ef3ef.slice/crio-f411e0a4d4953720aec4712c0012f28c5ee2beb2ddc7d720ba6cc6c4efb56717 WatchSource:0}: Error finding container f411e0a4d4953720aec4712c0012f28c5ee2beb2ddc7d720ba6cc6c4efb56717: Status 404 returned error can't find the container with id f411e0a4d4953720aec4712c0012f28c5ee2beb2ddc7d720ba6cc6c4efb56717
Sep 17 18:28:40.902505 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:40.902466    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerStarted","Data":"0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5"}
Sep 17 18:28:40.902701 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:40.902537    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerStarted","Data":"f411e0a4d4953720aec4712c0012f28c5ee2beb2ddc7d720ba6cc6c4efb56717"}
Sep 17 18:28:41.909910 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:41.909876    2585 generic.go:356] "Generic (PLEG): container finished" podID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerID="0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5" exitCode=0
Sep 17 18:28:41.910393 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:41.909937    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerDied","Data":"0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5"}
Sep 17 18:28:42.916974 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:42.916939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerStarted","Data":"efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834"}
Sep 17 18:28:42.917363 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:42.917003    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerStarted","Data":"fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc"}
Sep 17 18:28:42.917363 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:42.917029    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:42.937234 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:42.937186    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podStartSLOduration=2.9277085449999998 podStartE2EDuration="3.937174267s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.009465722s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:42.936032365 +0000 UTC m=+2856.127370076" watchObservedRunningTime="2026-09-17 18:28:42.937174267 +0000 UTC m=+2856.128511952"
Sep 17 18:28:43.921685 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:43.921653    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:43.923078 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:43.923049    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:28:44.926243 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:44.926205    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:28:49.931967 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:49.931898    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:28:49.932364 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:49.932280    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:28:59.933267 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:28:59.933228    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:29:09.933153 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:29:09.933116    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:29:19.932738 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:29:19.932694    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:29:29.932439 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:29:29.932397    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:29:39.932858 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:29:39.932797    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:29:49.932354 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:29:49.932316    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:29:59.932979 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:29:59.932947    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:30:09.875144 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:09.875108    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"]
Sep 17 18:30:09.875627 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:09.875348    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container" containerID="cri-o://fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc" gracePeriod=30
Sep 17 18:30:09.875627 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:09.875368    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kube-rbac-proxy" containerID="cri-o://efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834" gracePeriod=30
Sep 17 18:30:09.927497 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:09.927452    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 17 18:30:10.333907 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.333801    2585 generic.go:356] "Generic (PLEG): container finished" podID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerID="efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834" exitCode=2
Sep 17 18:30:10.333907 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.333873    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerDied","Data":"efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834"}
Sep 17 18:30:10.972587 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.972551    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"]
Sep 17 18:30:10.976362 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.976340    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:10.979020 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.978988    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 18:30:10.979136 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.979113    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:30:10.985796 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:10.985772    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"]
Sep 17 18:30:11.102585 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.102548    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwnf\" (UniqueName: \"kubernetes.io/projected/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-api-access-xcwnf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.102743 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.102590    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.102743 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.102626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.102743 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.102658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.203265 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.203225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwnf\" (UniqueName: \"kubernetes.io/projected/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-api-access-xcwnf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.203265 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.203266    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.203511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.203286    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.203511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.203307    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.203511 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:11.203432    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:30:11.203511 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:11.203511    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls podName:f3c866be-c57f-4e00-93d9-80f0a6e7bbd0 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:11.703490621 +0000 UTC m=+2944.894828288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" (UID: "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:30:11.203746 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.203720    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.204025 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.204006    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.212043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.212017    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwnf\" (UniqueName: \"kubernetes.io/projected/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-api-access-xcwnf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.708379 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.708342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.711073 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.711049    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:11.887639 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:11.887597    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:12.161091 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:12.161068    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"]
Sep 17 18:30:12.162709 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:30:12.162682    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c866be_c57f_4e00_93d9_80f0a6e7bbd0.slice/crio-9ab4766426c722a50dbd06ba85711724f53b4ab69efa0b12cb6a7ec10300138c WatchSource:0}: Error finding container 9ab4766426c722a50dbd06ba85711724f53b4ab69efa0b12cb6a7ec10300138c: Status 404 returned error can't find the container with id 9ab4766426c722a50dbd06ba85711724f53b4ab69efa0b12cb6a7ec10300138c
Sep 17 18:30:12.344956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:12.344922    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" event={"ID":"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0","Type":"ContainerStarted","Data":"ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92"}
Sep 17 18:30:12.345130 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:12.344993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" event={"ID":"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0","Type":"ContainerStarted","Data":"9ab4766426c722a50dbd06ba85711724f53b4ab69efa0b12cb6a7ec10300138c"}
Sep 17 18:30:14.527250 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.527222    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:30:14.633019 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.632993    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kserve-provision-location\") pod \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") "
Sep 17 18:30:14.633197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d787c22-368f-4ebb-92f4-8d44601ef3ef-proxy-tls\") pod \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") "
Sep 17 18:30:14.633197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633107    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq8lj\" (UniqueName: \"kubernetes.io/projected/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-api-access-jq8lj\") pod \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") "
Sep 17 18:30:14.633197 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633156    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-rbac-proxy-sar-config\") pod \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") "
Sep 17 18:30:14.633350 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633199    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-cabundle-cert\") pod \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\" (UID: \"3d787c22-368f-4ebb-92f4-8d44601ef3ef\") "
Sep 17 18:30:14.633350 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633291    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d787c22-368f-4ebb-92f4-8d44601ef3ef" (UID: "3d787c22-368f-4ebb-92f4-8d44601ef3ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:14.633495 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633470    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d787c22-368f-4ebb-92f4-8d44601ef3ef" (UID: "3d787c22-368f-4ebb-92f4-8d44601ef3ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:14.633602 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633553    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:14.633602 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633574    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:14.633602 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.633573    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d787c22-368f-4ebb-92f4-8d44601ef3ef" (UID: "3d787c22-368f-4ebb-92f4-8d44601ef3ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:14.635309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.635292    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d787c22-368f-4ebb-92f4-8d44601ef3ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d787c22-368f-4ebb-92f4-8d44601ef3ef" (UID: "3d787c22-368f-4ebb-92f4-8d44601ef3ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:14.635373 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.635324    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-api-access-jq8lj" (OuterVolumeSpecName: "kube-api-access-jq8lj") pod "3d787c22-368f-4ebb-92f4-8d44601ef3ef" (UID: "3d787c22-368f-4ebb-92f4-8d44601ef3ef"). InnerVolumeSpecName "kube-api-access-jq8lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:14.734860 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.734835    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d787c22-368f-4ebb-92f4-8d44601ef3ef-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:14.734860 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.734858    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d787c22-368f-4ebb-92f4-8d44601ef3ef-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:14.734993 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:14.734866    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8lj\" (UniqueName: \"kubernetes.io/projected/3d787c22-368f-4ebb-92f4-8d44601ef3ef-kube-api-access-jq8lj\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:15.360582 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.360492    2585 generic.go:356] "Generic (PLEG): container finished" podID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerID="fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc" exitCode=0
Sep 17 18:30:15.360582 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.360562    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"
Sep 17 18:30:15.360767 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.360574    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerDied","Data":"fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc"}
Sep 17 18:30:15.360767 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.360636    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2" event={"ID":"3d787c22-368f-4ebb-92f4-8d44601ef3ef","Type":"ContainerDied","Data":"f411e0a4d4953720aec4712c0012f28c5ee2beb2ddc7d720ba6cc6c4efb56717"}
Sep 17 18:30:15.360767 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.360661    2585 scope.go:117] "RemoveContainer" containerID="efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834"
Sep 17 18:30:15.371556 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.371533    2585 scope.go:117] "RemoveContainer" containerID="fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc"
Sep 17 18:30:15.379540 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.379522    2585 scope.go:117] "RemoveContainer" containerID="0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5"
Sep 17 18:30:15.380396 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.380376    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"]
Sep 17 18:30:15.383836 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.383799    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-5k7v2"]
Sep 17 18:30:15.387532 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.387509    2585 scope.go:117] "RemoveContainer" containerID="efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834"
Sep 17 18:30:15.387795 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:15.387770    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834\": container with ID starting with efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834 not found: ID does not exist" containerID="efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834"
Sep 17 18:30:15.387882 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.387799    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834"} err="failed to get container status \"efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834\": rpc error: code = NotFound desc = could not find container \"efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834\": container with ID starting with efd7d81eb667dedc7e602cb888b89e889ed33597383ac6066baa7147d45d1834 not found: ID does not exist"
Sep 17 18:30:15.387882 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.387842    2585 scope.go:117] "RemoveContainer" containerID="fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc"
Sep 17 18:30:15.388120 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:15.388101    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc\": container with ID starting with fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc not found: ID does not exist" containerID="fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc"
Sep 17 18:30:15.388169 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.388125    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc"} err="failed to get container status \"fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc\": rpc error: code = NotFound desc = could not find container \"fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc\": container with ID starting with fc3c323db430724551e10748f00a8d14c4fb002979e76fe3b0d02d9448c1fdfc not found: ID does not exist"
Sep 17 18:30:15.388169 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.388139    2585 scope.go:117] "RemoveContainer" containerID="0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5"
Sep 17 18:30:15.388380 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:15.388363    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5\": container with ID starting with 0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5 not found: ID does not exist" containerID="0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5"
Sep 17 18:30:15.388421 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:15.388388    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5"} err="failed to get container status \"0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5\": rpc error: code = NotFound desc = could not find container \"0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5\": container with ID starting with 0f99a2f895c636f960401e38bc2b82be979eef47304d18a0d7d6ee10b82acfc5 not found: ID does not exist"
Sep 17 18:30:16.962047 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:16.960308    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" path="/var/lib/kubelet/pods/3d787c22-368f-4ebb-92f4-8d44601ef3ef/volumes"
Sep 17 18:30:18.377375 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:18.377348    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr_f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/storage-initializer/0.log"
Sep 17 18:30:18.377701 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:18.377386    2585 generic.go:356] "Generic (PLEG): container finished" podID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerID="ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92" exitCode=1
Sep 17 18:30:18.377701 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:18.377443    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" event={"ID":"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0","Type":"ContainerDied","Data":"ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92"}
Sep 17 18:30:19.383880 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:19.383852    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr_f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/storage-initializer/0.log"
Sep 17 18:30:19.384230 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:19.383925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" event={"ID":"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0","Type":"ContainerStarted","Data":"0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65"}
Sep 17 18:30:20.962715 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:20.962687    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"]
Sep 17 18:30:20.963170 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:20.962864    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer" containerID="cri-o://0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65" gracePeriod=30
Sep 17 18:30:23.609798 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.609777    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr_f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/storage-initializer/1.log"
Sep 17 18:30:23.610216 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.610201    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr_f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/storage-initializer/0.log"
Sep 17 18:30:23.612203 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.612189    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:23.704778 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.704694    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kserve-provision-location\") pod \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") "
Sep 17 18:30:23.704778 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.704753    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-rbac-proxy-sar-config\") pod \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") "
Sep 17 18:30:23.705028 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.704792    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls\") pod \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") "
Sep 17 18:30:23.705028 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.704862    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcwnf\" (UniqueName: \"kubernetes.io/projected/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-api-access-xcwnf\") pod \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\" (UID: \"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0\") "
Sep 17 18:30:23.705028 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.704974    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" (UID: "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:23.705147 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.705082    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:23.705147 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.705082    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" (UID: "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:23.707073 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.707054    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" (UID: "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:23.707132 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.707093    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-api-access-xcwnf" (OuterVolumeSpecName: "kube-api-access-xcwnf") pod "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" (UID: "f3c866be-c57f-4e00-93d9-80f0a6e7bbd0"). InnerVolumeSpecName "kube-api-access-xcwnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:23.805888 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.805861    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:23.806017 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.805904    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:23.806017 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:23.805915    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwnf\" (UniqueName: \"kubernetes.io/projected/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0-kube-api-access-xcwnf\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:30:24.409926 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.409901    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr_f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/storage-initializer/1.log"
Sep 17 18:30:24.410317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.410301    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr_f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/storage-initializer/0.log"
Sep 17 18:30:24.410377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.410346    2585 generic.go:356] "Generic (PLEG): container finished" podID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerID="0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65" exitCode=1
Sep 17 18:30:24.410422 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.410397    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"
Sep 17 18:30:24.410457 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.410430    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" event={"ID":"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0","Type":"ContainerDied","Data":"0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65"}
Sep 17 18:30:24.410500 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.410489    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr" event={"ID":"f3c866be-c57f-4e00-93d9-80f0a6e7bbd0","Type":"ContainerDied","Data":"9ab4766426c722a50dbd06ba85711724f53b4ab69efa0b12cb6a7ec10300138c"}
Sep 17 18:30:24.410539 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.410517    2585 scope.go:117] "RemoveContainer" containerID="0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65"
Sep 17 18:30:24.420007 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.419989    2585 scope.go:117] "RemoveContainer" containerID="ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92"
Sep 17 18:30:24.427744 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.427729    2585 scope.go:117] "RemoveContainer" containerID="0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65"
Sep 17 18:30:24.428004 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:24.427986    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65\": container with ID starting with 0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65 not found: ID does not exist" containerID="0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65"
Sep 17 18:30:24.428060 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.428012    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65"} err="failed to get container status \"0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65\": rpc error: code = NotFound desc = could not find container \"0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65\": container with ID starting with 0434e6fb16b7b1b173685b8f1966db3829f69c345aa63302b6a885e113e27b65 not found: ID does not exist"
Sep 17 18:30:24.428060 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.428029    2585 scope.go:117] "RemoveContainer" containerID="ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92"
Sep 17 18:30:24.428298 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:30:24.428281    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92\": container with ID starting with ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92 not found: ID does not exist" containerID="ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92"
Sep 17 18:30:24.428351 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.428303    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92"} err="failed to get container status \"ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92\": rpc error: code = NotFound desc = could not find container \"ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92\": container with ID starting with ab4c0543ff0037fe980bca010ab2a4fe2b72576664c181da862432f1812bbf92 not found: ID does not exist"
Sep 17 18:30:24.446260 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.446234    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"]
Sep 17 18:30:24.448950 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.448928    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-v2bdr"]
Sep 17 18:30:24.960161 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:30:24.960130    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" path="/var/lib/kubelet/pods/f3c866be-c57f-4e00-93d9-80f0a6e7bbd0/volumes"
Sep 17 18:31:53.208122 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208042    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"]
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208422    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208439    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208452    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208458    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208471    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208480    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208499    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208508    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208520    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kube-rbac-proxy"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208528    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kube-rbac-proxy"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208617    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kserve-container"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208632    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer"
Sep 17 18:31:53.208654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208645    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d787c22-368f-4ebb-92f4-8d44601ef3ef" containerName="kube-rbac-proxy"
Sep 17 18:31:53.209145 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.208778    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c866be-c57f-4e00-93d9-80f0a6e7bbd0" containerName="storage-initializer"
Sep 17 18:31:53.211866 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.211844    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.214631 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.214608    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:31:53.214741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.214679    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:31:53.214741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.214678    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:31:53.214741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.214729    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 18:31:53.214741 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.214721    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:31:53.215058 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.215042    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:31:53.221971 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.221948    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"]
Sep 17 18:31:53.276503 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.276477    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.276629 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.276533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.276629 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.276583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cln\" (UniqueName: \"kubernetes.io/projected/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-api-access-95cln\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.276734 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.276633    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9934042b-e7be-44d1-aaf1-6272ad20a658-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.377511 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.377485    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cln\" (UniqueName: \"kubernetes.io/projected/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-api-access-95cln\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.377631 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.377525    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9934042b-e7be-44d1-aaf1-6272ad20a658-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.377631 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.377582    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.377631 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.377610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.377801 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:31:53.377710    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 18:31:53.377801 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:31:53.377765    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls podName:9934042b-e7be-44d1-aaf1-6272ad20a658 nodeName:}" failed. No retries permitted until 2026-09-17 18:31:53.877746513 +0000 UTC m=+3047.069084179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" (UID: "9934042b-e7be-44d1-aaf1-6272ad20a658") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 18:31:53.378065 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.378047    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9934042b-e7be-44d1-aaf1-6272ad20a658-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.378295 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.378278    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.388318 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.388292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cln\" (UniqueName: \"kubernetes.io/projected/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-api-access-95cln\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.881365 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.881332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:53.883891 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:53.883871    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:54.123574 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:54.123537    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:31:54.393661 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:54.393635    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"]
Sep 17 18:31:54.396126 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:31:54.396099    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9934042b_e7be_44d1_aaf1_6272ad20a658.slice/crio-3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f WatchSource:0}: Error finding container 3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f: Status 404 returned error can't find the container with id 3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f
Sep 17 18:31:54.825935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:54.825847    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" event={"ID":"9934042b-e7be-44d1-aaf1-6272ad20a658","Type":"ContainerStarted","Data":"390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a"}
Sep 17 18:31:54.825935 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:54.825910    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" event={"ID":"9934042b-e7be-44d1-aaf1-6272ad20a658","Type":"ContainerStarted","Data":"3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f"}
Sep 17 18:31:57.841061 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:57.840973    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/0.log"
Sep 17 18:31:57.841061 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:57.841014    2585 generic.go:356] "Generic (PLEG): container finished" podID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerID="390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a" exitCode=1
Sep 17 18:31:57.841061 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:57.841054    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" event={"ID":"9934042b-e7be-44d1-aaf1-6272ad20a658","Type":"ContainerDied","Data":"390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a"}
Sep 17 18:31:58.847088 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:58.847058    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/0.log"
Sep 17 18:31:58.847506 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:31:58.847183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" event={"ID":"9934042b-e7be-44d1-aaf1-6272ad20a658","Type":"ContainerStarted","Data":"e4a3f475acae27aa83fe760b848c63337826a3eb721c246fe33a11a5a6cb5581"}
Sep 17 18:32:01.860737 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.860709    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/1.log"
Sep 17 18:32:01.861173 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.861150    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/0.log"
Sep 17 18:32:01.861220 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.861182    2585 generic.go:356] "Generic (PLEG): container finished" podID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerID="e4a3f475acae27aa83fe760b848c63337826a3eb721c246fe33a11a5a6cb5581" exitCode=1
Sep 17 18:32:01.861279 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.861255    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" event={"ID":"9934042b-e7be-44d1-aaf1-6272ad20a658","Type":"ContainerDied","Data":"e4a3f475acae27aa83fe760b848c63337826a3eb721c246fe33a11a5a6cb5581"}
Sep 17 18:32:01.861336 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.861324    2585 scope.go:117] "RemoveContainer" containerID="390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a"
Sep 17 18:32:01.861640 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.861607    2585 scope.go:117] "RemoveContainer" containerID="390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a"
Sep 17 18:32:01.872453 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:32:01.872433    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_kserve-ci-e2e-test(9934042b-e7be-44d1-aaf1-6272ad20a658)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658"
Sep 17 18:32:01.872626 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:32:01.872596    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_kserve-ci-e2e-test_9934042b-e7be-44d1-aaf1-6272ad20a658_0 in pod sandbox 3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f from index: no such id: '390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a'" containerID="390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a"
Sep 17 18:32:01.872700 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:01.872631    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_kserve-ci-e2e-test_9934042b-e7be-44d1-aaf1-6272ad20a658_0 in pod sandbox 3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f from index: no such id: '390d2393771cd55b96473411c1e820c59bc93b17c34ca24a7d0250c1c50c312a'"
Sep 17 18:32:02.868348 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:02.868318    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/1.log"
Sep 17 18:32:03.190592 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.190515    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"]
Sep 17 18:32:03.321937 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.321915    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/1.log"
Sep 17 18:32:03.325299 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.325284    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:32:03.356377 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356354    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9934042b-e7be-44d1-aaf1-6272ad20a658-kserve-provision-location\") pod \"9934042b-e7be-44d1-aaf1-6272ad20a658\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") "
Sep 17 18:32:03.356501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356390    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-rbac-proxy-sar-config\") pod \"9934042b-e7be-44d1-aaf1-6272ad20a658\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") "
Sep 17 18:32:03.356501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356428    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls\") pod \"9934042b-e7be-44d1-aaf1-6272ad20a658\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") "
Sep 17 18:32:03.356501 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356447    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95cln\" (UniqueName: \"kubernetes.io/projected/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-api-access-95cln\") pod \"9934042b-e7be-44d1-aaf1-6272ad20a658\" (UID: \"9934042b-e7be-44d1-aaf1-6272ad20a658\") "
Sep 17 18:32:03.356681 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356581    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9934042b-e7be-44d1-aaf1-6272ad20a658-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9934042b-e7be-44d1-aaf1-6272ad20a658" (UID: "9934042b-e7be-44d1-aaf1-6272ad20a658"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:03.356681 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356662    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9934042b-e7be-44d1-aaf1-6272ad20a658-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:32:03.356850 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.356799    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9934042b-e7be-44d1-aaf1-6272ad20a658" (UID: "9934042b-e7be-44d1-aaf1-6272ad20a658"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:03.358558 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.358540    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9934042b-e7be-44d1-aaf1-6272ad20a658" (UID: "9934042b-e7be-44d1-aaf1-6272ad20a658"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:03.358674 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.358658    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-api-access-95cln" (OuterVolumeSpecName: "kube-api-access-95cln") pod "9934042b-e7be-44d1-aaf1-6272ad20a658" (UID: "9934042b-e7be-44d1-aaf1-6272ad20a658"). InnerVolumeSpecName "kube-api-access-95cln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:03.457317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.457258    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:32:03.457317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.457281    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9934042b-e7be-44d1-aaf1-6272ad20a658-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:32:03.457317 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.457290    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95cln\" (UniqueName: \"kubernetes.io/projected/9934042b-e7be-44d1-aaf1-6272ad20a658-kube-api-access-95cln\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:32:03.880509 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.880480    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx_9934042b-e7be-44d1-aaf1-6272ad20a658/storage-initializer/1.log"
Sep 17 18:32:03.880973 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.880579    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"
Sep 17 18:32:03.880973 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.880582    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx" event={"ID":"9934042b-e7be-44d1-aaf1-6272ad20a658","Type":"ContainerDied","Data":"3518a6273d7e378c8a04f7aa094d8e4ecbaab6a50140cb2ab8633e0eb296a34f"}
Sep 17 18:32:03.880973 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.880680    2585 scope.go:117] "RemoveContainer" containerID="e4a3f475acae27aa83fe760b848c63337826a3eb721c246fe33a11a5a6cb5581"
Sep 17 18:32:03.918229 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.918200    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"]
Sep 17 18:32:03.921779 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:03.921756    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-mplrx"]
Sep 17 18:32:04.346830 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.346726    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"]
Sep 17 18:32:04.347133 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.347118    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerName="storage-initializer"
Sep 17 18:32:04.347133 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.347135    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerName="storage-initializer"
Sep 17 18:32:04.347265 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.347152    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerName="storage-initializer"
Sep 17 18:32:04.347265 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.347158    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerName="storage-initializer"
Sep 17 18:32:04.347265 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.347219    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerName="storage-initializer"
Sep 17 18:32:04.347265 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.347228    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" containerName="storage-initializer"
Sep 17 18:32:04.351884 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.351856    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.354529 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354500    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:32:04.354662 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354500    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:32:04.354662 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354625    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:32:04.354662 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354628    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:04.354662 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354643    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:32:04.354930 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354917    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:32:04.354974 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.354962    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 18:32:04.361195 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.361169    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"]
Sep 17 18:32:04.462662 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.462634    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.462794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.462671    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.462794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.462694    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8zb\" (UniqueName: \"kubernetes.io/projected/6f44dae3-32ab-484a-8767-0962931cd95f-kube-api-access-gv8zb\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.462794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.462754    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f44dae3-32ab-484a-8767-0962931cd95f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.462794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.462776    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f44dae3-32ab-484a-8767-0962931cd95f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.563686 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.563659    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f44dae3-32ab-484a-8767-0962931cd95f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.563831 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.563714    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.563882 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.563852    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.563926 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.563895    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8zb\" (UniqueName: \"kubernetes.io/projected/6f44dae3-32ab-484a-8767-0962931cd95f-kube-api-access-gv8zb\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.563974 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.563924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f44dae3-32ab-484a-8767-0962931cd95f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.564284 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.564262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f44dae3-32ab-484a-8767-0962931cd95f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.564432 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.564385    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.564432 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.564423    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.566287 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.566262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f44dae3-32ab-484a-8767-0962931cd95f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.572043 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.572002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8zb\" (UniqueName: \"kubernetes.io/projected/6f44dae3-32ab-484a-8767-0962931cd95f-kube-api-access-gv8zb\") pod \"isvc-pmml-runtime-predictor-885fc8777-vqvfl\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.663743 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.663720    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:04.934427 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.934396    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"]
Sep 17 18:32:04.936249 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:32:04.936223    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f44dae3_32ab_484a_8767_0962931cd95f.slice/crio-9ddfcfa889648b40cbd474b4e0753e5512f4ef51aed822c3ded65edfafd49269 WatchSource:0}: Error finding container 9ddfcfa889648b40cbd474b4e0753e5512f4ef51aed822c3ded65edfafd49269: Status 404 returned error can't find the container with id 9ddfcfa889648b40cbd474b4e0753e5512f4ef51aed822c3ded65edfafd49269
Sep 17 18:32:04.960990 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:04.960968    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9934042b-e7be-44d1-aaf1-6272ad20a658" path="/var/lib/kubelet/pods/9934042b-e7be-44d1-aaf1-6272ad20a658/volumes"
Sep 17 18:32:05.893182 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:05.893134    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerStarted","Data":"386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0"}
Sep 17 18:32:05.893397 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:05.893201    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerStarted","Data":"9ddfcfa889648b40cbd474b4e0753e5512f4ef51aed822c3ded65edfafd49269"}
Sep 17 18:32:08.915052 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:08.915021    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f44dae3-32ab-484a-8767-0962931cd95f" containerID="386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0" exitCode=0
Sep 17 18:32:08.915490 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:08.915095    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerDied","Data":"386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0"}
Sep 17 18:32:08.916463 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:08.916446    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:32:16.968280 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:16.968244    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerStarted","Data":"66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f"}
Sep 17 18:32:16.968657 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:16.968307    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerStarted","Data":"edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345"}
Sep 17 18:32:16.968657 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:16.968480    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:16.987892 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:16.987854    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podStartSLOduration=1.822425991 podStartE2EDuration="12.987842455s" totalImagesPullingTime="7.374751037s" totalInitContainerRuntime="3.790665427s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:16.986362962 +0000 UTC m=+3070.177700648" watchObservedRunningTime="2026-09-17 18:32:16.987842455 +0000 UTC m=+3070.179180141"
Sep 17 18:32:17.639281 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:17.639255    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:32:17.642009 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:17.641983    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:32:17.976342 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:17.976257    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:17.977574 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:17.977550    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:32:18.980617 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:18.980575    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:32:23.984798 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:23.984759    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:32:23.985464 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:23.985435    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:32:33.985555 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:33.985509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:32:43.986312 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:43.986268    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:32:53.985498 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:32:53.985448    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:33:03.985950 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:03.985914    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:33:13.985546 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:13.985504    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:33:23.985975 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:23.985888    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:33:33.986383 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:33.986332    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 17 18:33:36.622434 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:36.622402    2585 scope.go:117] "RemoveContainer" containerID="0032a1fd454f04d0fd859d4dda96e3c4a1088ac8ff0ac721c446e242497afd50"
Sep 17 18:33:36.630722 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:36.630703    2585 scope.go:117] "RemoveContainer" containerID="5cfde61f68f21ac39f9bf4570bd6a03195c54f732d81afafa60f6c807c287123"
Sep 17 18:33:36.638163 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:36.638143    2585 scope.go:117] "RemoveContainer" containerID="7b452f7ea806802f3ef114ee680dc9328cef2f703802635b36da457f64aa1bd6"
Sep 17 18:33:43.986911 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:43.986884    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:33:45.923308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:45.923273    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"]
Sep 17 18:33:45.923779 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:45.923540    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container" containerID="cri-o://edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345" gracePeriod=30
Sep 17 18:33:45.923779 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:45.923609    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kube-rbac-proxy" containerID="cri-o://66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f" gracePeriod=30
Sep 17 18:33:46.390707 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:46.390666    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f44dae3-32ab-484a-8767-0962931cd95f" containerID="66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f" exitCode=2
Sep 17 18:33:46.390893 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:46.390732    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerDied","Data":"66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f"}
Sep 17 18:33:48.980722 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:48.980678    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 17 18:33:49.862615 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.862592    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:33:49.982477 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982445    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-kube-rbac-proxy-sar-config\") pod \"6f44dae3-32ab-484a-8767-0962931cd95f\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") "
Sep 17 18:33:49.982956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982489    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f44dae3-32ab-484a-8767-0962931cd95f-proxy-tls\") pod \"6f44dae3-32ab-484a-8767-0962931cd95f\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") "
Sep 17 18:33:49.982956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982512    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv8zb\" (UniqueName: \"kubernetes.io/projected/6f44dae3-32ab-484a-8767-0962931cd95f-kube-api-access-gv8zb\") pod \"6f44dae3-32ab-484a-8767-0962931cd95f\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") "
Sep 17 18:33:49.982956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982531    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-cabundle-cert\") pod \"6f44dae3-32ab-484a-8767-0962931cd95f\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") "
Sep 17 18:33:49.982956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982570    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f44dae3-32ab-484a-8767-0962931cd95f-kserve-provision-location\") pod \"6f44dae3-32ab-484a-8767-0962931cd95f\" (UID: \"6f44dae3-32ab-484a-8767-0962931cd95f\") "
Sep 17 18:33:49.982956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982916    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f44dae3-32ab-484a-8767-0962931cd95f" (UID: "6f44dae3-32ab-484a-8767-0962931cd95f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:49.982956 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982930    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f44dae3-32ab-484a-8767-0962931cd95f" (UID: "6f44dae3-32ab-484a-8767-0962931cd95f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:49.983170 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.982965    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f44dae3-32ab-484a-8767-0962931cd95f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f44dae3-32ab-484a-8767-0962931cd95f" (UID: "6f44dae3-32ab-484a-8767-0962931cd95f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:49.984718 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.984699    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f44dae3-32ab-484a-8767-0962931cd95f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f44dae3-32ab-484a-8767-0962931cd95f" (UID: "6f44dae3-32ab-484a-8767-0962931cd95f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:49.984831 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:49.984766    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f44dae3-32ab-484a-8767-0962931cd95f-kube-api-access-gv8zb" (OuterVolumeSpecName: "kube-api-access-gv8zb") pod "6f44dae3-32ab-484a-8767-0962931cd95f" (UID: "6f44dae3-32ab-484a-8767-0962931cd95f"). InnerVolumeSpecName "kube-api-access-gv8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:50.083309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.083253    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f44dae3-32ab-484a-8767-0962931cd95f-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:33:50.083309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.083274    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:33:50.083309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.083283    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f44dae3-32ab-484a-8767-0962931cd95f-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:33:50.083309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.083293    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv8zb\" (UniqueName: \"kubernetes.io/projected/6f44dae3-32ab-484a-8767-0962931cd95f-kube-api-access-gv8zb\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:33:50.083309 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.083302    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f44dae3-32ab-484a-8767-0962931cd95f-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:33:50.412654 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.412624    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f44dae3-32ab-484a-8767-0962931cd95f" containerID="edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345" exitCode=0
Sep 17 18:33:50.412855 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.412688    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"
Sep 17 18:33:50.412855 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.412711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerDied","Data":"edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345"}
Sep 17 18:33:50.412855 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.412770    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl" event={"ID":"6f44dae3-32ab-484a-8767-0962931cd95f","Type":"ContainerDied","Data":"9ddfcfa889648b40cbd474b4e0753e5512f4ef51aed822c3ded65edfafd49269"}
Sep 17 18:33:50.412855 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.412799    2585 scope.go:117] "RemoveContainer" containerID="66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f"
Sep 17 18:33:50.422173 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.422141    2585 scope.go:117] "RemoveContainer" containerID="edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345"
Sep 17 18:33:50.429595 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.429579    2585 scope.go:117] "RemoveContainer" containerID="386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0"
Sep 17 18:33:50.435375 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.435355    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"]
Sep 17 18:33:50.436899 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.436873    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-vqvfl"]
Sep 17 18:33:50.438004 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.437991    2585 scope.go:117] "RemoveContainer" containerID="66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f"
Sep 17 18:33:50.438269 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:33:50.438250    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f\": container with ID starting with 66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f not found: ID does not exist" containerID="66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f"
Sep 17 18:33:50.438336 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.438281    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f"} err="failed to get container status \"66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f\": rpc error: code = NotFound desc = could not find container \"66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f\": container with ID starting with 66cd6e6cd3b4a556715f1da769d88ce78d12e2bdd44540111ed46f52b05b5e4f not found: ID does not exist"
Sep 17 18:33:50.438336 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.438307    2585 scope.go:117] "RemoveContainer" containerID="edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345"
Sep 17 18:33:50.438565 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:33:50.438549    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345\": container with ID starting with edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345 not found: ID does not exist" containerID="edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345"
Sep 17 18:33:50.438632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.438574    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345"} err="failed to get container status \"edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345\": rpc error: code = NotFound desc = could not find container \"edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345\": container with ID starting with edea0072f1ca201db0618f2007e32b97f8e59219832bfffb85d95c3a8940d345 not found: ID does not exist"
Sep 17 18:33:50.438632 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.438594    2585 scope.go:117] "RemoveContainer" containerID="386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0"
Sep 17 18:33:50.438856 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:33:50.438832    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0\": container with ID starting with 386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0 not found: ID does not exist" containerID="386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0"
Sep 17 18:33:50.438917 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.438865    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0"} err="failed to get container status \"386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0\": rpc error: code = NotFound desc = could not find container \"386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0\": container with ID starting with 386472b2b868654408849428e8b3cc00e9c2c66b17807f34d97e4211fb0873a0 not found: ID does not exist"
Sep 17 18:33:50.959628 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:33:50.959589    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" path="/var/lib/kubelet/pods/6f44dae3-32ab-484a-8767-0962931cd95f/volumes"
Sep 17 18:35:22.311549 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311516    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"]
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311936    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="storage-initializer"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311951    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="storage-initializer"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311965    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311973    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311990    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kube-rbac-proxy"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.311996    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kube-rbac-proxy"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.312078    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kserve-container"
Sep 17 18:35:22.312156 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.312089    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f44dae3-32ab-484a-8767-0962931cd95f" containerName="kube-rbac-proxy"
Sep 17 18:35:22.314268 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.314252    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.316850 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.316822    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:35:22.317001 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.316975    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:35:22.317001 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.316990    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-6a05e5\""
Sep 17 18:35:22.318088 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.318018    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6a05e5-predictor-serving-cert\""
Sep 17 18:35:22.318088 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.318045    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-6a05e5-dockercfg-lv57w\""
Sep 17 18:35:22.318088 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.318054    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:35:22.318088 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.318061    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6a05e5-kube-rbac-proxy-sar-config\""
Sep 17 18:35:22.324611 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.324591    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"]
Sep 17 18:35:22.398134 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.398102    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kserve-provision-location\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.398308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.398156    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.398308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.398183    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.398308 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.398207    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7zc\" (UniqueName: \"kubernetes.io/projected/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-api-access-vd7zc\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.398457 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.398307    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-cabundle-cert\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499116 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kserve-provision-location\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499252 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499132    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499252 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499164    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499252 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7zc\" (UniqueName: \"kubernetes.io/projected/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-api-access-vd7zc\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499415 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-cabundle-cert\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499415 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:35:22.499319    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-serving-cert: secret "isvc-secondary-6a05e5-predictor-serving-cert" not found
Sep 17 18:35:22.499415 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:35:22.499392    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls podName:fe16c523-6fc7-478e-90cc-e7104f4ad0f5 nodeName:}" failed. No retries permitted until 2026-09-17 18:35:22.999372094 +0000 UTC m=+3256.190709760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls") pod "isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" (UID: "fe16c523-6fc7-478e-90cc-e7104f4ad0f5") : secret "isvc-secondary-6a05e5-predictor-serving-cert" not found
Sep 17 18:35:22.499575 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499476    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kserve-provision-location\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499845 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499795    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.499937 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.499919    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-cabundle-cert\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:22.507984 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:22.507958    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7zc\" (UniqueName: \"kubernetes.io/projected/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-api-access-vd7zc\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:23.003490 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:23.003451    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:23.005911 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:23.005890    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls\") pod \"isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") " pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:23.226477 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:23.226438    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:23.496789 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:23.496761    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"]
Sep 17 18:35:23.498526 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:35:23.498498    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe16c523_6fc7_478e_90cc_e7104f4ad0f5.slice/crio-c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71 WatchSource:0}: Error finding container c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71: Status 404 returned error can't find the container with id c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71
Sep 17 18:35:23.836957 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:23.836870    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" event={"ID":"fe16c523-6fc7-478e-90cc-e7104f4ad0f5","Type":"ContainerStarted","Data":"c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8"}
Sep 17 18:35:23.836957 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:23.836945    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" event={"ID":"fe16c523-6fc7-478e-90cc-e7104f4ad0f5","Type":"ContainerStarted","Data":"c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71"}
Sep 17 18:35:30.869998 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:30.869968    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/0.log"
Sep 17 18:35:30.870444 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:30.870006    2585 generic.go:356] "Generic (PLEG): container finished" podID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerID="c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8" exitCode=1
Sep 17 18:35:30.870444 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:30.870031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" event={"ID":"fe16c523-6fc7-478e-90cc-e7104f4ad0f5","Type":"ContainerDied","Data":"c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8"}
Sep 17 18:35:31.877675 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:31.877647    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/0.log"
Sep 17 18:35:31.878059 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:31.877703    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" event={"ID":"fe16c523-6fc7-478e-90cc-e7104f4ad0f5","Type":"ContainerStarted","Data":"3611e666608f416414e4d506e582e34ef09ac5e7f56159dcd0ffaf6b37e2b29b"}
Sep 17 18:35:33.888472 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.888444    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/1.log"
Sep 17 18:35:33.888914 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.888897    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/0.log"
Sep 17 18:35:33.888964 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.888932    2585 generic.go:356] "Generic (PLEG): container finished" podID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerID="3611e666608f416414e4d506e582e34ef09ac5e7f56159dcd0ffaf6b37e2b29b" exitCode=1
Sep 17 18:35:33.889000 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.888967    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" event={"ID":"fe16c523-6fc7-478e-90cc-e7104f4ad0f5","Type":"ContainerDied","Data":"3611e666608f416414e4d506e582e34ef09ac5e7f56159dcd0ffaf6b37e2b29b"}
Sep 17 18:35:33.889035 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.889026    2585 scope.go:117] "RemoveContainer" containerID="c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8"
Sep 17 18:35:33.889409 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.889389    2585 scope.go:117] "RemoveContainer" containerID="c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8"
Sep 17 18:35:33.899702 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:35:33.899670    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_kserve-ci-e2e-test(fe16c523-6fc7-478e-90cc-e7104f4ad0f5)\"" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5"
Sep 17 18:35:33.899979 ip-10-0-138-144 kubenswrapper[2585]: E0917 18:35:33.899953    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_kserve-ci-e2e-test_fe16c523-6fc7-478e-90cc-e7104f4ad0f5_0 in pod sandbox c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71 from index: no such id: 'c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8'" containerID="c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8"
Sep 17 18:35:33.900046 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:33.899986    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_kserve-ci-e2e-test_fe16c523-6fc7-478e-90cc-e7104f4ad0f5_0 in pod sandbox c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71 from index: no such id: 'c23964704f425b683cdfed60ecee515e4fb592b5fb94139586c7ff9e4f07d8b8'"
Sep 17 18:35:34.895440 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:34.895407    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/1.log"
Sep 17 18:35:42.316787 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.316752    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"]
Sep 17 18:35:42.448286 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.448266    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/1.log"
Sep 17 18:35:42.450577 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.450561    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:42.564553 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564515    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-cabundle-cert\") pod \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") "
Sep 17 18:35:42.564553 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564563    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-rbac-proxy-sar-config\") pod \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") "
Sep 17 18:35:42.564766 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564581    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd7zc\" (UniqueName: \"kubernetes.io/projected/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-api-access-vd7zc\") pod \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") "
Sep 17 18:35:42.564870 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564762    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls\") pod \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") "
Sep 17 18:35:42.564870 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564797    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kserve-provision-location\") pod \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\" (UID: \"fe16c523-6fc7-478e-90cc-e7104f4ad0f5\") "
Sep 17 18:35:42.564980 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564927    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe16c523-6fc7-478e-90cc-e7104f4ad0f5" (UID: "fe16c523-6fc7-478e-90cc-e7104f4ad0f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:42.564980 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.564939    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe16c523-6fc7-478e-90cc-e7104f4ad0f5" (UID: "fe16c523-6fc7-478e-90cc-e7104f4ad0f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:42.565160 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.565133    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-cabundle-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:35:42.565261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.565166    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:35:42.565261 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.565138    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe16c523-6fc7-478e-90cc-e7104f4ad0f5" (UID: "fe16c523-6fc7-478e-90cc-e7104f4ad0f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:42.566993 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.566939    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe16c523-6fc7-478e-90cc-e7104f4ad0f5" (UID: "fe16c523-6fc7-478e-90cc-e7104f4ad0f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:42.566993 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.566952    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-api-access-vd7zc" (OuterVolumeSpecName: "kube-api-access-vd7zc") pod "fe16c523-6fc7-478e-90cc-e7104f4ad0f5" (UID: "fe16c523-6fc7-478e-90cc-e7104f4ad0f5"). InnerVolumeSpecName "kube-api-access-vd7zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:42.666094 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.666067    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-proxy-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:35:42.666094 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.666090    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kserve-provision-location\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:35:42.666248 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.666103    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7zc\" (UniqueName: \"kubernetes.io/projected/fe16c523-6fc7-478e-90cc-e7104f4ad0f5-kube-api-access-vd7zc\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Sep 17 18:35:42.933617 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.933594    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72_fe16c523-6fc7-478e-90cc-e7104f4ad0f5/storage-initializer/1.log"
Sep 17 18:35:42.933830 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.933664    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72" event={"ID":"fe16c523-6fc7-478e-90cc-e7104f4ad0f5","Type":"ContainerDied","Data":"c30e1f2310b91f6e0cc9f1ffccab9d3461b8ad044ce69c544060364222336c71"}
Sep 17 18:35:42.933830 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.933699    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"
Sep 17 18:35:42.933830 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.933714    2585 scope.go:117] "RemoveContainer" containerID="3611e666608f416414e4d506e582e34ef09ac5e7f56159dcd0ffaf6b37e2b29b"
Sep 17 18:35:42.975794 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.975764    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"]
Sep 17 18:35:42.977457 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:42.977427    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6a05e5-predictor-5f5d4d69bc-2vz72"]
Sep 17 18:35:44.958987 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:35:44.958944    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" path="/var/lib/kubelet/pods/fe16c523-6fc7-478e-90cc-e7104f4ad0f5/volumes"
Sep 17 18:36:28.700962 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:28.700892    2585 ???:1] "http: TLS handshake error from 10.0.131.184:47072: EOF"
Sep 17 18:36:28.708093 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:28.708071    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6g6jp_fa7b3894-1a3f-441c-80a0-7ff6ae96b749/global-pull-secret-syncer/0.log"
Sep 17 18:36:28.895537 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:28.895509    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pr9b6_750bb16e-a510-490e-8b0f-285c9ea415ac/konnectivity-agent/0.log"
Sep 17 18:36:28.996092 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:28.996021    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-144.ec2.internal_acff49d59889ccaa0427b1a9c3b732ea/haproxy/0.log"
Sep 17 18:36:32.484778 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.484754    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/alertmanager/0.log"
Sep 17 18:36:32.518619 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.518596    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/config-reloader/0.log"
Sep 17 18:36:32.546385 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.546363    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/kube-rbac-proxy-web/0.log"
Sep 17 18:36:32.569760 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.569743    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/kube-rbac-proxy/0.log"
Sep 17 18:36:32.602361 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.602339    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/kube-rbac-proxy-metric/0.log"
Sep 17 18:36:32.630732 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.630707    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/prom-label-proxy/0.log"
Sep 17 18:36:32.663982 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.663963    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61d9c50-e55f-4b8d-9f94-035b088e918a/init-config-reloader/0.log"
Sep 17 18:36:32.771754 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.771694    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-8785r_fdb8f115-24a5-4243-a106-3da5baf1f912/kube-state-metrics/0.log"
Sep 17 18:36:32.809293 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.809278    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-8785r_fdb8f115-24a5-4243-a106-3da5baf1f912/kube-rbac-proxy-main/0.log"
Sep 17 18:36:32.847826 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:32.847788    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-8785r_fdb8f115-24a5-4243-a106-3da5baf1f912/kube-rbac-proxy-self/0.log"
Sep 17 18:36:33.062585 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.062518    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86mjv_de540b4a-d1bb-474f-b327-a9c7b0cd67a5/node-exporter/0.log"
Sep 17 18:36:33.095543 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.095522    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86mjv_de540b4a-d1bb-474f-b327-a9c7b0cd67a5/kube-rbac-proxy/0.log"
Sep 17 18:36:33.120524 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.120499    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86mjv_de540b4a-d1bb-474f-b327-a9c7b0cd67a5/init-textfile/0.log"
Sep 17 18:36:33.227638 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.227614    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-4jcgh_e8f2cd82-a372-4402-907f-21ee068db39d/kube-rbac-proxy-main/0.log"
Sep 17 18:36:33.254394 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.254372    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-4jcgh_e8f2cd82-a372-4402-907f-21ee068db39d/kube-rbac-proxy-self/0.log"
Sep 17 18:36:33.283560 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.283537    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-4jcgh_e8f2cd82-a372-4402-907f-21ee068db39d/openshift-state-metrics/0.log"
Sep 17 18:36:33.608162 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.608134    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d9d47dcb-7jkns_85521a94-1d73-4718-8b4b-a71da71cb442/telemeter-client/0.log"
Sep 17 18:36:33.631185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.631162    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d9d47dcb-7jkns_85521a94-1d73-4718-8b4b-a71da71cb442/reload/0.log"
Sep 17 18:36:33.658653 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.658633    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d9d47dcb-7jkns_85521a94-1d73-4718-8b4b-a71da71cb442/kube-rbac-proxy/0.log"
Sep 17 18:36:33.689934 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.689915    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/thanos-query/0.log"
Sep 17 18:36:33.720593 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.720572    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy-web/0.log"
Sep 17 18:36:33.743994 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.743974    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy/0.log"
Sep 17 18:36:33.769424 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.769406    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/prom-label-proxy/0.log"
Sep 17 18:36:33.794301 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.794277    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy-rules/0.log"
Sep 17 18:36:33.818454 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:33.818434    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bdbbcdc9-6w5m9_3e309d14-c79c-4ea2-b258-319bca8c92c2/kube-rbac-proxy-metrics/0.log"
Sep 17 18:36:36.217655 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:36.217625    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-zr5kx_5008d57f-0a28-4cfe-a37f-08667f53ea6e/download-server/0.log"
Sep 17 18:36:36.765295 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:36.765273    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-pzxc8_202532de-c9e0-4705-9e6c-3a8252b95e44/volume-data-source-validator/0.log"
Sep 17 18:36:37.145766 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.145729    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"]
Sep 17 18:36:37.146185 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.146165    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerName="storage-initializer"
Sep 17 18:36:37.146278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.146188    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerName="storage-initializer"
Sep 17 18:36:37.146278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.146221    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerName="storage-initializer"
Sep 17 18:36:37.146278 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.146229    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerName="storage-initializer"
Sep 17 18:36:37.146453 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.146323    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerName="storage-initializer"
Sep 17 18:36:37.146453 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.146335    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe16c523-6fc7-478e-90cc-e7104f4ad0f5" containerName="storage-initializer"
Sep 17 18:36:37.150671 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.150651    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.153235 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.153215    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bc5r6\"/\"kube-root-ca.crt\""
Sep 17 18:36:37.154398 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.154377    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bc5r6\"/\"default-dockercfg-mqmws\""
Sep 17 18:36:37.154484 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.154440    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bc5r6\"/\"openshift-service-ca.crt\""
Sep 17 18:36:37.157954 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.157933    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"]
Sep 17 18:36:37.169726 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.169696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-lib-modules\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.170037 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.169912    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nqf\" (UniqueName: \"kubernetes.io/projected/d249bda4-c24a-414b-bde1-442736e33eb3-kube-api-access-n2nqf\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.170037 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.169968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-podres\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.170037 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.169997    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-sys\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.170241 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.170096    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-proc\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.270579 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nqf\" (UniqueName: \"kubernetes.io/projected/d249bda4-c24a-414b-bde1-442736e33eb3-kube-api-access-n2nqf\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.270579 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270590    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-podres\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270606    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-sys\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270637    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-proc\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270663    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-lib-modules\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-podres\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-proc\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270755    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-sys\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.271105 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.270792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d249bda4-c24a-414b-bde1-442736e33eb3-lib-modules\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.278762 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.278736    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nqf\" (UniqueName: \"kubernetes.io/projected/d249bda4-c24a-414b-bde1-442736e33eb3-kube-api-access-n2nqf\") pod \"perf-node-gather-daemonset-7sl4w\" (UID: \"d249bda4-c24a-414b-bde1-442736e33eb3\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.461544 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.461463    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:37.729725 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.729701    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"]
Sep 17 18:36:37.731789 ip-10-0-138-144 kubenswrapper[2585]: W0917 18:36:37.731756    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd249bda4_c24a_414b_bde1_442736e33eb3.slice/crio-b8d6d472443175b8cb5b88ac6426422647f9650c0c2e5d525026fd42970988ba WatchSource:0}: Error finding container b8d6d472443175b8cb5b88ac6426422647f9650c0c2e5d525026fd42970988ba: Status 404 returned error can't find the container with id b8d6d472443175b8cb5b88ac6426422647f9650c0c2e5d525026fd42970988ba
Sep 17 18:36:37.797494 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.797476    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2nm9_5792e032-4987-4024-984f-950180feebc9/dns/0.log"
Sep 17 18:36:37.824927 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.824904    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2nm9_5792e032-4987-4024-984f-950180feebc9/kube-rbac-proxy/0.log"
Sep 17 18:36:37.869311 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:37.869288    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nvkjm_9e6f7109-7114-4a3e-922f-408eb432d5dd/dns-node-resolver/0.log"
Sep 17 18:36:38.194034 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:38.194001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w" event={"ID":"d249bda4-c24a-414b-bde1-442736e33eb3","Type":"ContainerStarted","Data":"322689d8ad44361e02c6bd3a743e0d242779d325af84b9ec32b6c9c30c4428e6"}
Sep 17 18:36:38.194204 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:38.194191    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:38.194254 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:38.194224    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w" event={"ID":"d249bda4-c24a-414b-bde1-442736e33eb3","Type":"ContainerStarted","Data":"b8d6d472443175b8cb5b88ac6426422647f9650c0c2e5d525026fd42970988ba"}
Sep 17 18:36:38.211590 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:38.211542    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w" podStartSLOduration=1.21152795 podStartE2EDuration="1.21152795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:38.209705247 +0000 UTC m=+3331.401042932" watchObservedRunningTime="2026-09-17 18:36:38.21152795 +0000 UTC m=+3331.402865634"
Sep 17 18:36:38.364757 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:38.364730    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-795764c55d-qhqkl_a4bc3f6a-8dce-46c6-b793-947b2469763e/registry/0.log"
Sep 17 18:36:38.389770 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:38.389752    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kggnz_2177cca8-a58d-4b81-8a10-65f67ee8ada0/node-ca/0.log"
Sep 17 18:36:39.563926 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:39.563889    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjxg6_800607ce-206f-4ce9-8edc-d6636d819654/serve-healthcheck-canary/0.log"
Sep 17 18:36:40.061300 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:40.061271    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbcjh_1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9/kube-rbac-proxy/0.log"
Sep 17 18:36:40.082061 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:40.082034    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbcjh_1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9/exporter/0.log"
Sep 17 18:36:40.109065 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:40.109042    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbcjh_1c0491fc-6fa7-48ef-9fac-85bc0e06a6f9/extractor/0.log"
Sep 17 18:36:42.376680 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:42.376644    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5c5cdf4877-4qgtw_688356f1-2037-4d78-bd3d-93e037a91c57/manager/0.log"
Sep 17 18:36:43.187287 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:43.187238    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-4p625_4334a790-ec3b-42e2-8c64-b6831ea3aa5f/seaweedfs/0.log"
Sep 17 18:36:43.235416 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:43.235390    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-qf62g_ac9556ce-5089-4792-9983-79819b5824a0/seaweedfs-tls-serving/0.log"
Sep 17 18:36:44.210326 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:44.210294    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-7sl4w"
Sep 17 18:36:47.802942 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:47.802917    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-892xx_29df95df-3880-4cf6-9de8-caa826016278/kube-storage-version-migrator-operator/1.log"
Sep 17 18:36:47.803980 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:47.803963    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-892xx_29df95df-3880-4cf6-9de8-caa826016278/kube-storage-version-migrator-operator/0.log"
Sep 17 18:36:49.152227 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.152202    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:36:49.174436 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.174407    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/egress-router-binary-copy/0.log"
Sep 17 18:36:49.195567 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.195548    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/cni-plugins/0.log"
Sep 17 18:36:49.218642 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.218619    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/bond-cni-plugin/0.log"
Sep 17 18:36:49.239414 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.239356    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/routeoverride-cni/0.log"
Sep 17 18:36:49.260241 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.260220    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/whereabouts-cni-bincopy/0.log"
Sep 17 18:36:49.280885 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.280862    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbqb_db93ed24-9b8e-49c4-96a4-164a1d955bb5/whereabouts-cni/0.log"
Sep 17 18:36:49.353226 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.353200    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-phph5_19e0e20a-4fbb-4b22-bfc0-d08736b8fd4c/kube-multus/0.log"
Sep 17 18:36:49.405686 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.405618    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29686_a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5/network-metrics-daemon/0.log"
Sep 17 18:36:49.426167 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:49.426121    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29686_a842e3d3-81ce-4a5a-bbd5-12edef8ff6e5/kube-rbac-proxy/0.log"
Sep 17 18:36:50.467666 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.467637    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-controller/0.log"
Sep 17 18:36:50.486101 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.486078    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/1.log"
Sep 17 18:36:50.486222 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.486086    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovn-acl-logging/0.log"
Sep 17 18:36:50.507159 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.507135    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/kube-rbac-proxy-node/0.log"
Sep 17 18:36:50.528048 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.528022    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:36:50.549392 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.549375    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/northd/0.log"
Sep 17 18:36:50.570327 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.570308    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/nbdb/0.log"
Sep 17 18:36:50.591289 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.591236    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/sbdb/0.log"
Sep 17 18:36:50.635019 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:50.634992    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g9d4s_3075df7b-db2c-4495-8264-1a9102b2fe4b/ovnkube-controller/0.log"
Sep 17 18:36:51.858600 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:51.858573    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kt64l_066cdf30-b263-4044-970e-46a388a81e48/network-check-target-container/0.log"
Sep 17 18:36:52.731153 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:52.731118    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7pvw9_4f77808b-f0e6-4982-8dc4-6839a9fca62b/iptables-alerter/0.log"
Sep 17 18:36:53.462033 ip-10-0-138-144 kubenswrapper[2585]: I0917 18:36:53.462001    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-j5nt8_b1601798-247c-4826-b3ba-c2b0b26ab562/tuned/0.log"